This project's evaluation of NDPs will use a comparative, cross-national analysis to place each country's experiences within a broader international context, and to explain both the particular features of a country's policy and more universal aspects of policy-making (Reich, 1988). Cross-national studies of health policy in developing countries are relatively few in number. In the late 1970s. Montgomery, Lasswell, and Migdal published a book on population policies, which used comparative and longitudinal approaches to analyze patterns of policies (Montgomery, et al., 1979). Their study explicitly examined both the processes of policy making and implementation as well as the substance of population policy, and thereby represents a potential model for this project. A recent book on the policies and politics of AIDS in the industrialized countries represents another effort to use a comparative approach to health policy from both substantive and procedural perspectives (Kirp et al., 1992). There are a few publications of collected essays on pharmaceutical problems in developing countries (Robles, et al., 1992; Geest, et al., 1988). No studies, however, specifically focus on pharmaceutical policy, and the analyses of pharmaceutical policy that exist rarely explore the problems of policy performance and implementation in a systematic manner. This project, thus, will represent a significant advance in the empirical application of comparative policy analysis in developing countries.
The three methods of standardized indicators, group evaluation, and political mapping are described next.
1. Performance Indicators
Purpose of Indicators: The standardized indicators represent a tool for summative and formative evaluation (see Background Paper). The proposed indicators serve two primary purposes:
a) To evaluate the outcomes of pharmaceutical policies in a comparative framework; and
b) To assess pharmaceutical policy formulation and implementation (i.e. policy process).
The outcomes of the NDPs will be measured according to the four common goals of NDPs in developing countries: availability, accessibility, good quality and rational use of drugs. In addition, other country-specific goals may be added to the evaluation. The assessment of success will be based on the extent to which countries have been able to achieve the precise goals and targets (initial or evolving) that they set for themselves (summative evaluation). In addition, the evaluation will examine the processes of policy formulation and implementation, and the institutional and political contexts in which the policies were implemented (formative evaluation).
Selecting the Indicators: The indicators for this comparative evaluation are selected from the indicators developed in a recent manual for monitoring national drug policies in developing countries (Brudon-Jakobowicz, Rainhorn and Reich, 1994). The indicators are classified as general data indicators, structural indicators, process indicators, and outcome indicators.
These are supplemented by some new indicators (in the form of standardized questions) based on hypotheses from the Background Paper for assessing relevant aspects of NDP formulation and implementation. The draft indicator manual, which lists the indicators, their purpose, description, and the methods of data collection, is provided as Appendix 2. A summary description of the project's indicators is provided next.
Description of Indicators: The policy formulation and implementation indicators (standardized questions) are based on hypotheses generated in the Background Paper, and focus on the process of pharmaceutical policy formulation. These indicators will supplement the political mapping methodology in assessing the organizational and political factors that affect the successful formulation and implementation of pharmaceutical policy.
The policy performance indicators are derived from the manual developed by a collaboration between APED/WHO, the Harvard School of Public Health, and the Centre de Recherches and d'Etudes pour le Développement de la Santé (Paris) (Brudon-Jakobowicz, et al., 1994), which followed nine logical steps:
1. Literature Review to identify potential key issues and components of pharmaceutical policy in developing countries.
2. Delphi Survey to develop consensus on key issues and components of pharmaceutical policy in developing countries.
3. Experts Consultation to review general difficulties in indicator development and define criteria for selection of indicators.
4. Working Group to propose sets of indicators for monitoring implementation of drug policy in developing countries.
5. Field Testing of proposed indicators in six countries to assess the clarity, applicability, and usefulness of the indicators selected.
6. Review of the first draft manual by experts within and outside WHO to assess the methodologies used for indicator development and the categories of indicators.
7. Working group to propose a set of outcome indicators to measure the progress toward the overall objectives.
8. Review of methodology for indicator calculation by some epidemiologists and statisticians within and outside WHO to assess the relevance of the proposed methodologies and appropriateness of the sampling procedures.
9. Finalization of the manual based on a review of all comments received and incorporation of appropriate revisions.
The four categories of indicators are briefly described:
Background information are intended to provide data on the demographic, health, economic, and pharmaceutical contexts in which drug policy is being implemented in a given country. These indicators are quantitative data, at a single point in time, and in most countries are readily available at the central level. Some of the information will be used in calculating subsequent indicators. Some data may be useful in cross-national comparisons of drug policy implementation.
Structural indicators provide qualitative information to assess the pharmaceutical system's capacity to achieve its policy objectives. These indicators are intended to check if the key structures necessary to implement a pharmaceutical policy are present in the country. Seven categories of key components are considered: legislation and regulation, essential drug selection and drug registration, drug allocation in the health budget/public sector financing policies, public sector procurement procedures, public sector distribution and logistics, pricing policy, information and continuous education on drug use. The indicators can be used in comparing the pharmaceutical policies of different countries and in assisting national and international decision-makers to design interventions to improve the pharmaceutical sector.
Process indicators provide quantitative measurements of the processes by which a national drug policy is implemented. The indicators assess the degree to which activities are being effectively implemented and the progress over time. The process indicator monitor the main aspects under the same seven key components of drug policy. These indicators can be used for assisting national decision-makers in monitoring progress in the implementation of the drug policy in relation to goals and targets set up at the national level.
Outcome indicators are intended to measure the results achieved and the changes that can be attributed to the implementation of a national drug policy. These indicators measure the effects of the policy on the overall objectives: availability and affordability of essential drugs, drug quality, and the rational use of drugs. These indicators can be used in assisting national and international decision-makers in measuring progress towards overall objectives and adjusting strategies accordingly, and in comparing pharmaceutical policies of different countries.
Implementation of Indicators: The application of these indicators will be carried out by the respective country teams. The methodology for calculating the indicator, including a detailed discussion of the procedures for conducting surveys, as well as potential sources for the required information, are provided in the indicator manual (Appendix 2).
2. Group Evaluation
Purpose of Group Evaluation: The group evaluation will be carried out in two rounds, with two main purposes: a) To assist in the initiation of the field research by training the participants in the indicator and political mapping research methodologies, and to achieve group consensus on a common approach [Round 1]. b) To review the accomplishments and conclusions of the application of the indicators and the political mapping, in a comparative framework, and in a critical but supportive group process; and to identify specific questions of possible interest to all developing countries implementing NDPs, for further investigation [Round 2].
Description of Group Evaluation: Group evaluation will involve the following activities:
1) training the participants in the two research methods and achieving group consensus on the application;
2) comparing the available policy performance indicators for all countries;
3) discussing the performance methodologies and proposing adjustments, with particular consideration of its potential incorporation into national health policy information systems;
4) discussing cross-national comparisons, and the extent to which the methodologies successfully accomplished this objective;
5) examining proposed explanations for relative strengths and weaknesses of pharmaceutical policy performance in different countries;
6) identifying specific questions relevant to NDP implementation, which could be usefully investigated in some countries in a second round of political mapping; and
7) reviewing policy interventions that could improve pharmaceutical policy performance in each participating country.
Implementation of Group Evaluation: Round 1: The first round of group evaluation will occur after each country team has performed a preliminary analysis of the indicators, in preparation for the full application of the indicators and political mapping. This meeting is scheduled for October, 1994.
Round 2: The second round of group evaluation will occur after the full application of the indicators and the political mapping. Each country team will prepare an evaluation of pharmaceutical policy performance, including: 1) the results of the performance indicators and the political mapping exercise, 2) an evaluation of the performance methodology, and 3) a document that proposes explanations for strengths and weaknesses shown by the indicators and the political mapping. These three sets of materials will be critically reviewed by the participants from the three collaborating institutions, and representatives from the country teams. Questions to be investigated in certain countries will also be identified.
3. Political Mapping
Purpose of Mapping: Political mapping has three main goals for this project:
a) To assess the organizational and political determinants of the policy process;
b) To help explain the strengths and weaknesses identified by the performance indicators; and
c) To assist in the identifying and assessing strategies to improve pharmaceutical policy implementation.
Description of the political mapping process: Political mapping constitutes the third method of the research, and provides an analysis of the process of NDP formulation and implementation. The overall objective of political mapping is to assist decision-makers in improving the implementation of health policies.
This methodology has descriptive, explanatory, and prescriptive objectives: to describe the consequences, stakeholders, interests, and networks involved in a particular policy; to help explain how and why a particular decision was reached in the past; and to assist decision-makers in managing the politics of formulation or implementation (Reich. 1993).
Political mapping involves six discrete steps, presented in detail in the mapping manual (Appendix 3):
Policy Consequences: describes the policy's consequences along four dimensions: the size, identity, intensity, and timing of the effects.
Position Map: provides a map of the proponents and opponents of the health policy, according to international sector, political sector, government sector, private sector, nongovernmental sector, and social sector.
Stakeholder Analysis: determines the main interests and objectives of each organization, and the organizational priority of the interests.
Policy Network Analysis: identifies the formal and informal linkages between organizations and individuals.
Transitions Assessment: identifies major transitions underway in the organization responsible for implementing the policy, the general organizational environment, and the broader political environment.
Strategies for Change: analyzes potential strategies for changing or influencing policy-making processes and outcomes according to symbolic. positive, and negative strategies.
This methodology is currently under development at the Harvard School of Public Health. It has been applied to a series of health policy problems, and has been shown to provide rapid identification of problems, to improve communication among organizations, and to assist policy-makers in identifying effective new strategies for policy implementation.
Implementation of Political Mapping: The political mapping will be undertaken by each country team, assisted by the three collaborating institutions. The mapping will be based on published documents as well as non-published documents (including internal government memoranda) and interviews with key policy-makers in the stakeholder organizations in both public and private sectors. The mapping process, thus, will require active interaction with policy-makers responsible for pharmaceutical policy. It is expected that the results of the political mapping will be helpful to policy-makers in designing effective strategies to improve implementation. Detailed guidelines on implementing the mapping exercise are provided in Appendix 3.
A second round may be instituted in some countries, in order to pursue questions identified by the group evaluation as particularly germane to NDP formulation or implementation in developing countries.