(2003; 158 pages)
Chapter 4: Methods for monitoring and evaluating processes and outcomes
Monique F. Mrazek and Elias Mossialos
1. The notion of monitoring
When any form of policy - economic or otherwise - is intended to have an impact on pharmaceutical care, it is important to measure its intended (and possibly unintended) effects. The present chapter will consider methods that can be used to monitor those effects on both process and outcomes. Monitoring process involves a regular review of the activities that make up drug management and delivery programs, and that are intended to achieve policy objectives as regards to both health and expenditure; it is sometimes more clearly known as “internal” assessment. Monitoring the results, i.e. the outcomes, will show whether these policy objectives are being achieved; this is sometimes termed “external” assessment.
Monitoring initially involves selecting particular indicators that can serve as measures of policy performance. Indicators of both process and outcomes are initially measured and then followed over the course of time. The information obtained can be compared against a predetermined target for each indicator providing a basis for the evaluation.
2. Linking monitoring to policy goals and objectives
2.1. Components of policy
Any long-term policy needs to have a clear overall goal, as well as a set of specific objectives which one intends to attain and a list of the outcomes which can be anticipated. These three components need to be set in advance when the policy is planned and adopted. One will also need to devise a system to measure progress as the policy is implemented.
The ultimate goal of drug policies must be to improve the well-being and health status of the patient population. However, given the inevitable constraints on funding in this area, an associated goal would be to ensure that drugs are managed, delivered and used in a cost-effective manner. A competing goal in setting drug policy may be to foster or sustain the contribution of the R&D based drug industry to the domestic economy. Where a balance is struck between competing health and industrial policy goals will depend on the weightings of the domestic interests.
The objectives of a policy are detailed end-points that need to be attained and steps that need to be taken over time. Objectives are commonly defined by an analysis of the population’s clinical needs and demands to be met.
Finally, one will have to define at the outset the outcomes that can realistically be anticipated, i.e. the specific effects in the field, at the level where the community is affected by the policy.
The monitoring of policies will thus need to be based not only on internal feedback from within the administrative structure (to check on measures taken and other aspects of implementation), but also on external feedback from the field (to determine the attainment of objectives and the effects as the population experiences them). Monitoring will have to reflect all aspects of the execution of the policy, but in particular the goals, objectives and outcomes which have been defined in advance, should wherever possible, be concrete enough to render possible the measurement of progress. The methods of measurement are considered under Sections 3 and 4, below.
While all these concepts are straightforward, one can run into problems when putting them into practice. It might for example be said that the level and pattern of drug utilization is an outcome measure, but quite apart from the difficulties which can arise when measuring utilization (see Chapter 2) it may be far from clear what its significance is. The level of drug use does not necessarily equal the level of clinical need, since there may be over-use or under-use as well as inappropriate patterns of use. Nor is the level of clinical need necessarily an indication of the level of demand, for as pointed out earlier in this volume the latter may be unreasonably high and sometimes (e.g., because of lack of money) unreasonably low. Levels of utilization, clinical need and demand therefore should be tracked separately. Each of them varies from country to country according to epidemiological, demographic and cultural factors, while the availability of resources (such as personnel, technology, institutions and finance) to meet needs will vary between health care systems. It follows that the goals and objectives of drug policies are likely to be country- and health care system-specific, and that the measures of progress that can be used in those policies will similarly be specific to the situation.
2.2. Setting priorities
Because some of the various interests reflected in a policy may inevitably conflict with one another - especially in terms of vying for financial resources - priorities will have be set at the outset. Criteria that are well recognized for evaluating the allocation of health care and pharmaceutical resources are effectiveness, efficiency, equity and quality. As the policy is implemented, the feedback attained through monitoring may indicate the need to revise or adjust the policy or some of its components, and this can mean a rethinking of priorities and a reallocation of resources. For example if a cost containment plan for drugs proves to be producing disproportionate hardship for the elderly (inequity) or causes a worsening of outcomes (ineffectiveness), or even increases costs rather than reducing them (which will point to increasing inefficiency unless there has been no commensurate increase in effectiveness) these problems should show up in the course of monitoring and the priorities may have to be revised.
2.3. Setting objectives
The point has already been made that progress towards meeting objectives must wherever possible be measurable and it is therefore necessary that objectives should be expressed in concrete rather than general terms . Objectives should be set at both micro and macro levels. Micro level objectives concern effects on individual patients, physicians or institutions. At the macro level objectives will typically relate to the operation of the pharmaceutical service, the overall health care system or the wider social, political and economic environment in which government decisions are taken; examples of the latter might include the objective of securing a change in the system of university medical education or amendments in the tax system relating to drug imports.
2.4. Effectiveness as a criterion
Drug cost containment policies should not significantly impair effectiveness in terms of improvements in health outcomes, and wherever possible effectiveness should be enhanced . Better health outcomes should be expected if the process of drug use is made more effective. Whether drug management and delivery truly has become more effective will depend on the quantity, quality and appropriateness of the drugs prescribed, dispensed and consumed, relative to a patient’s clinical need. This is true at both the micro and macro levels. Patient needs can be assessed using a consensus of expert opinion as to what ought-to-be provided, based on the clinical facts and the epidemiological situation. Improvements in the effectiveness of the process of drug management and delivery can be detected in terms of health outcomes in the patient population.
2.5. Efficiency as a criterion
Efficiency relates health outcomes to the resources used to produce them. Drug policies should aim to maximize health gains subject to equity constraints (i.e. equity-efficiency trade-off) while keeping the costs to society as low as they can reasonably be. At the macro level it is important to achieve both production efficiency (producing services at the least cost) and efficiency in the allocation and application of those resources (maximizing health gains given limited resources). Maximizing efficiency at a micro level implies among other things that effective pharmaceuticals should be supplied at a price reflecting the contribution of the drug to improving health status as compared with alternative treatments. In other words, this means maximizing the use of cost-effective drugs. Efficiency at all levels is thus a necessary element in improving economic outcomes.
2.6. Equity as a criterion
Equity is concerned with ensuring that the benefits and burdens of pharmaceutical care are fairly distributed. Applying the principle of equity is complex and can be confusing. Equity in terms of finance means that patients, society, the professions and the industry should each make a reasonable contribution to the cost, commensurate with their means. Distributing access and delivery fairly will involve ensuring that all population groups are treated equally, with no deprivation as a result of their location, age, sex or other characteristics.
It is simple to propound these general principles of equity, but in applying them one is inevitably faced with the difficulty of deciding, in specific situations, what is “just” and “fair”. If, to take an extreme example, a group of individuals live by choice or necessity on a remote outlying island, far removed from any pharmacy or specialised prescriber, will it be “fair” to expend the same percentage of our budget per household on them as on city dwellers, despite the fact that less resources can be purchased for the same amount of money for the islanders as compared to the city dwellers? Or should one, to be equitable, allocate more of our budget and hence resources to the islanders in order to attain an urban level of service, despite the fact that the city dwellers will in effect be shouldering a disproportionate burden to the benefits received?
Multiple definitions of a fair or just distribution have been used in health care, reflecting variously quality of utilisation, distribution according to need, equality of access and equality of health (see [6,16]). Each of these definitions relies on value judgements which differ (see ). When evaluating the effect of a drug policy in a particular health system it is therefore important to know which definition of equity has been adopted. A system adopting an egalitarian approach (distribution according to need and financing based on ability to pay) will have different objectives than a libertarian system (entitled to what they get provided it is acquired justly), and therefore cannot be evaluated using the same definition of equity.
2.7. Quality as a criterion
All the criteria so far discussed are certainly important but it is also crucial that what is provided meets the patient’s reasonable expectations with of quality at one level may be to equate it to effectiveness, but the definition of quality certainly goes beyond a reflection of how patients perceive the effectiveness of the services they receive. The definition of quality is also likely to differ depending on cultural expectations and values, as well as the nature of the health care system through which pharmaceutical services are delivered. For example, a US patient may equate quality to freedom of choice, while a patient in the UK may equate it to shorter waiting times or to reduced variations in services across the system. Expert definitions of quality include the dimensions of access, appropriateness, and technical and/or interpersonal excellence as measured by the health outcomes achieved [3,10]. Therefore access to pharmaceutical services (i.e. availability of prescribers, pharmacies and medicines) should be appropriate to what patients need and delivered in a manner that displays good levels of professional practice.
Minimum quality standards must certainly equate to the minimum standards governing licensing of professional health practitioners. Drugs for example may be overprescribed, or prescribed and used inappropriately suggesting poor quality of professional care. This may be associated with the risk of increased illness and again indicate a poor quality service. Monitoring quality may require that data be collected on the behaviour of individual prescribers which is certain to be met by some resistance particularly if individuals are to be singled out amongst their piers for poor performance. However, if the quality of individual professional performance is to be evaluated and changes made, than these efforts need to be done with the co-operation and participation of the professional staff or there is likely to be resistance to change .
Effectiveness, efficiency, equity and quality are naturally inter-linked. Improving clinical effectiveness (for example by increasing the use of medicines of proven effectiveness) may improve quality. Increasing effectiveness while controlling for costs may increase efficiency. In increasing efficiency, new opportunities can be created for improved effectiveness and equity. However, there may have to be trade-offs between these laudable ideals. For example, maximising effectiveness and efficiency will not necessarily result in a more equitable distribution of pharmaceutical resources or better quality pharmaceutical services. The costs which the community is prepared to assume may increase in order to improve quality, effectiveness and equity, but they may decrease if the community is determined to push for efficiency at the expense of other criteria. Therefore, it is important to find a balance between these different criteria.
3. Types of indicators
To determine the effect of drug policies one will have to find and select suitable indicators (i.e. measures). Indicators that will prove helpful in measuring the effects of a cost containment policy will relate variously to internal process (i.e. implementation of the policy) and to the policy’s actual effects on drug management and delivery in the field (external indicators) as measurable in terms of health and economics.
3.1. Internal (“process”) indicators
In the administration of health care programmes, both governments and hospitals routinely collect process data. Input variables include the numbers of doctors and pharmacists; output variables include the volume of drugs prescribed and dispensed. Measures of process generally relate a quantity of input to some meaningful denominator; commonly used denominators include population served (e.g., drugs dispensed per thousand population) or patient population actually served (e.g., costs per patient day).
To assess the efficiency of the process in the delivery of pharmaceuticals, indicators have been developed that relate cost along various dimensions including input or output. Pricing indexes are a measure of process that can be used by policy makers to monitor the change in drug prices over time, though the figures may be distorted by changes to the product basket (see Chapter 2). Process data can also be used to develop prescribing indicators to monitor whether drug choice is appropriate, effective, safe and economical; a national insurance fund is for example likely to have very detailed data on prescribing in individual practices and for individual patients. Effectiveness can be monitored using process indicators of quantity, quality and appropriateness associated with variations in drug utilisation data. Indicators of the use of pharmaceutical services, relative to patient need, can be used to assess equity in terms of access. Indicators of the process of drug delivery such as availability (distribution of prescribers and pharmacies), organisation (types of facilities) and financing (co-payment arrangements) can be used to monitor equity in terms of potential access and the freedom of choice.
Improvements in process indicators do not necessarily signal better health or economic outcomes. A given input does not guarantee a given output, and an output may be affected for better or for worse by factors quite divorced from the input. There may also be a considerable time lag between changes in process indicators and associated indicators of outcomes. For such reasons it is not sufficient to monitor policies solely in terms of progress in implementation on the one hand or observed outcomes on the other; both are needed.
3.2. Outcome indicators
As implied above, one will always need to test the ultimate effect of policies in terms of what actually happens in the field where they are supposed to take effect. For example, the cost-effectiveness of drug interventions and programmes can be assessed using economic evaluations. Health status should be measured both at the individual and population level. At the individual level health outcomes focus on measuring clinical or physiological changes. The health status of populations can be monitored using epidemiological data such as mortality, morbidity, disease prevalence and incidence, as well as measures of social and economic productivity . All the time one will need to beware of the pitfall noted under Paragraph 3.1. above, i.e., a known output may not have resulted from the known input, or from that factor alone.
4. Developing indicators for monitoring
Success in monitoring process and outcomes depends on the reliability of the indicators which are used. Five criteria should be considered when selecting indicators: usefulness, clarity, measurability, reliability and validity . Issues relevant to the quality of indicators will be briefly considered here.
First of all, the indicators developed should be reliable, practical and useful to decision makers, helping them to improve the processes and outcomes of pharmaceutical care. Concrete issues directly related to policies and for which suitable indicators have to be found include drug expenditure, consumption, prescribing and pricing, as well as health and economic outcomes. The indicators developed should be able to detect the effect of a policy in terms of effectiveness, efficiency and equity criteria. Some indicators, for example prescribing figures from a well-managed reimbursement system, are obviously reliable and relevant; others may well need to be field-tested and the results discussed with decision-makers.
An indicator must also be reasonably amenable to measurement. Just what one measures is likely to depend on the facilities one has, and which types of figures are most likely to be complete, representative, accurate, rapidly available and statistically valid . Some types of data can be (and often are) captured in large administrative data bases; however these, as pointed out in Chapter 2, may have been designed for uses other than cost containment and policy review, and lack precisely the variables which one now needs. Data from such an information system may therefore prove disappointing when one puts questions which the system was not designed to answer. If one has any reason to doubt the reliability of the data in such a system, some form of peer review at the source will be advisable, for example to ensure that data are being entered consistently and in line with prescribed procedures.
In a cost containment programme, quantified data (especially costs) are obviously likely to be of most use, but qualitative data can also be very helpful. Qualitative indicators are generally based on “yes” or “no” answers to survey questions and are particularly useful in monitoring attitudes and preferences. Prescribing can for example be monitored both quantitatively in terms of the number of prescriptions written but also qualitatively in terms of changes in physician knowledge or attitudes regarding the prescribing decision.
Special studies such as randomised control clinical trials or observational studies can be used to gather data that are not collected by routine monitoring systems; these however involve additional costs and it may prove very difficult to maintain them consistently over a long period. Sentinel reporting systems, which collect data on a carefully selected sample, are useful for monitoring the short-term impact of policy implementation, particularly in order to detect unexpected or unintended outcomes . Time pressures, the availability of financial resources and the willingness of staff members to comply with requirements will influence the type of monitoring method selected.
Establishing indicators for international comparisons is more difficult. As noted elsewhere in this chapter and earlier in this volume (Chapter 2), variability can be expected in the manner, types and extent of data collected between countries. In addition, ongoing policy implementation and reform can make it difficult to identify and maintain compatible indicators over a period of time. However, certain indicators for international comparisons of national drug policies  and indicators of drug use  have been defined and they are worth examining if it is proposed to compare policies across borders.
5. Setting monitoring targets
Just as each objective needs indicators, so each indicator needs targets in time. If for example one objective of a cost containment programme is to reduce the well-documented over-prescribing of antidepressant drugs, an indicator can be the quarterly prescribing level for five major drugs of this type as documented by national insurance payments, and targets could be a 20% reduction by December 2003 and a 50% reduction by December 2005.
Explicit targets for indicators should be incorporated at each stage of the policy, planning and budgeting process. At the policy stage, to take the same example as above, it may be decided that the over-prescribing of antidepressant drugs represents an unhealthy trend in mental health and brings with it heavy expenditure as well as a risk of dependence. At the planning stage, consideration of what is known about the true levels of clinical depression in the population may lead to the conclusion that prescribing at one third of the current level is desirable and attainable; examination of the national insurance records will confirm that these provide a reliable indicator for the turnover in the principal antidepressant drugs. The budget will explicitly incorporate a projection for reduced prescribing costs, but also provisions for professional and patient re-education regarding depression. The targets will then be set in terms of achievements anticipated by 2003 and 2005. Defining and setting clear indicator-linked targets at the beginning of the policy process thus helps to ensure the objectivity of evaluation in later stages.
6. Techniques for evaluating projects and programs
6.1. Economic evaluation
Economic evaluation is an important tool for assessing the micro efficiency and effectiveness of pharmaceutical programs in terms of costs and outcomes, as well as cost/effect relationships for individual medicines. In seeking to maximise both effectiveness and efficiency despite constraints on resources, governments and industry have increasingly used economic evaluations to compare alternative courses of action in terms of both their costs and health outcomes. Economic evaluations generally draw upon decision analytic models so that the cost and health consequences of alternative treatment and probable pathways can be simultaneously analysed. Decision trees are useful tools for calculating and comparing the expected cost per patient outcome of alternative treatment strategies. Using a decision tree, an analyst is able to model and compare the costs and outcomes of a medical intervention (e.g., drug therapy) against that of alternative forms of treatment and (or) other medical or surgical interventions. This type of evaluation can simultaneously take into account the alternative means of delivering pharmaceutical care in terms of method, place, timing or quantity, at all phases of treatment .
There are essentially three basic types of economic evaluation: cost-benefit analysis (CBA), cost-effectiveness analysis (CEA) and cost-utility analysis (CUA). The debate is still on going as to which of these approaches is to be preferred.
Cost-benefit analysis (CBA) measures both cost and benefits in monetary terms as it seeks to identify interventions that will bring about potential improvements in welfare. This technique involves the need to place a monetary value on items such as health benefit. A number of means of doing this exist (e.g., the human capital method or measures of an individual’s contingent valuation), but to some extent the figures are unavoidably artificial. For example, the human capital approach does not measure the individual’s . Although revealed preferences and contingent valuation do attempt to measure willingness-to-pay, the former involves an uncoupling of the valuation of consequences from the context , while the latter is still considered experimental.
Cost-effectiveness analysis (CEA) again measures costs in monetary units but it expresses effectiveness using a biological yet quantifiable unit of effect such as the number of lives saved or life years gained. CEA assesses the incremental costs and incremental effectiveness of one medical intervention relative to another. Unlike CBA, CEA ratios must be compared to an external standard and outcomes common to both alternatives are needed in order to judge relative desirability. A special form of CEA known as cost-minimisation analysis is essentially limited to comparing costs between alternative forms of treatment where the effectiveness of each has been demonstrated or can reasonably be assumed to be the same.
Cost-utility analysis (CUA), like CEA, is concerned with incremental analysis. CUA, however, quantifies effect in terms of “utility”. Utility in this sense represents subjective values (i.e. quality weights) given to their overall health status; over a period of time one may express the health status in “quality adjusted life years” (QALYs) or “health related quality of life (HRQL). These quality weights have the merit of combining both the positive and negative effects of a treatment into a single summary score that usually ranges between 0 (equal to death) and 1 (equal to perfect health). It is possible to compare health outcomes in different diseases using the elicited values. Combining health state utilities with cost information renders it possible to calculate cost-utility ratios for alternative treatments.
In spite of advances made in carrying out such measurements, the application of CUA is currently limited by problems associated with the determination of utility values and QALYs. In many instances, for example, utility scores cannot be obtained from patients themselves for such reasons as severity of illness, mental capacity or age, and they have to be derived by the observer from other groups in society. But this may be a good thing, as many observers believe that in systems where the public pays for health care, QALYs should be elicited from a random sample of the population.
Conducting an economic evaluation requires data on costs (i.e. unit costs or prices of resources), physical quantity of resources consumed (by the treatments being compared) and outcomes (i.e. defining comparative treatment effects and how treatment affects quality of live) . The basic data requirements for both health and economic outcomes were discussed in Chapter 2. The data needed to conduct an economic evaluation will depend on the perspective or viewpoint used to determine the range of costs. Commonly chosen perspectives for economic evaluations are those of the health care provider, the patient, the community and society generally. The societal perspective includes all costs born by patients, insurers or other parties, while the patient perspective includes only those costs born by the patient. Therefore, changing the viewpoint will affect the range of costs considered and could affect the choice of programme or treatment being evaluated.
The data needed also depends on the time frame used. The time frame used for collecting data on costs and outcomes should not be misleading and bias the analysis in favour of one intervention over another. If costs are followed over a lifetime then it is standard practice to discount these future costs to present values. Methodological debates surrounding economic evaluations and how they are used by decision-makers is discussed in Chapter 5.
6.2. Evaluating quality
Several methods exist for evaluating the quality of pharmaceutical care provided from prescribing through to delivery. The narrowest of these is the Drug Utilisation Review (DURs). DURs are used to evaluate the level and pattern of drug use. They were originally motivated by concerns about excessive or inappropriate prescribing . DURs can be used to document the association between inappropriate prescribing and its adverse clinical and/or economic consequences. Appropriateness of drug use can be assessed on three levels:
(i) whether any medication is warranted;
(ii) if indicated, which drug would be the agent of choice;
(iii) whether the chosen drug is used in an appropriate manner (dosage, duration, type and frequency of monitoring and the risk of drug interactions) .
However, it is not always easy to make such an objective assessment at any of these three levels. To determine acceptability one will have to set the identified pattern of use against some recognised standard which is applicable to that particular therapy (for example as defined in an authoritative therapeutic guideline). This can be time-consuming  and even controversial but necessary.
A second common method of evaluating quality is that of audit. Audit is used to determine whether an optimal quality of service is being achieved within the resources available. The evaluation can be used to uncover areas of problem or concern to prescribers and their patients, to determine whether there is an effective use of resources to achieve desired outcomes or to address equity considerations in terms of geographical, social or ethnic differences in service use and prescribing patterns in relation to population needs. There are two main types of audit; medical audit evaluates the quality of medical care, including procedures used for treatment, the use of resources and the resulting outcome and quality of life enjoyed by patients  while clinical audit focuses on the quality of professional care by doctors, nurses, pharmacists and other health care providers (e.g., in the delivery of pharmaceutical services). If the findings are indeed favourable, health providers and users can be reassured; if not, changes will be needed. Adjustments to service made in the light of medical audit findings can lead to improved patient outcomes, better quality of life and the more cost-effective use of resources. A prerequisite to audit is the availability of accurate patient information . The main disadvantage of audit is that it takes a considerable amount of money and professional time.
Finally, quality assurance provides a systematic approach to evaluating health and pharmaceutical care. A WHO working group  defined four components of quality assurance in health care: patient satisfaction with services provided; professional performance (technical quality); resource use (efficiency); and risk management (the risk of illness or injury associated with the service). Quality assurance evaluations of prescribing for example have focused variously on prescribing patterns to assess physician competence and appropriateness in prescribing (i.e. performance) , on drug utilisation to evaluate different strategies to contain costs (i.e. resource use) , on compliance with prescribing guidelines to monitor the continuum of quality use of medicines between in-patient and out-patient care , and on the monitoring of preventable adverse outcomes (risk management). Important in performing a quality assurance assessment will be appropriate and valid indicators as well as a meaningful time-frame. For example, one study that was evaluating adherence to prescribing guidelines found that indicators evaluated at an individual patient level give differing results to aggregate data, and self-reported instruments were found to overestimate adherence .
6.3. Production and cost models
The extent to which efficiency has been achieved can be assessed using production and cost models. These models apply production and cost functions to any production process; they determine the particular mix of inputs which will result in the lowest costs for any given level of output and the optimal size of the production if costs there are to be kept to a reasonable minimum. These models have been applied to physician, hospital and insurance services to determine the extent to which production efficiencies have been achieved and how they can be further improved .
6.4. International comparisons
In an attempt to gain a perspective on the macro efficiency of the drug system, helpful comparisons can be made to other politically and economically similar countries. One must however beware of possible pitfalls in making international comparisons because of cultural, demographic and political differences between countries, as well as, methodological issues associated with the data used in the comparisons; the matter has been previously discussed in Chapter 2 and is touched on again in Section 7 below.
7. Limitations of monitoring and the use of indicators
The use of monitoring and indicators is not without problems. Some specific pitfalls have been noted above, but there are also more general difficulties. While for example performance, as judged on the basis of a single indicator, may have improved, it is important to put this indicator back into the wider policy context and consider whether the overall policy change which brought about this improvement was on balance beneficial. It is particularly important to consider whether using the selected indicators may have introduced perverse incentives that shift problems from one area into another that is not monitored. Making an effort to meet indicator targets can mean that other important issues get neglected, a dilemma known as Goodhart’s Law of if some variable is made the focus of measuring policy performance, then people and institutions will devote a disproportionate amount of time and effort to meeting this target at the expense of other aspects of their performance which are not subject to the same level of scrutiny. The result can be disbalance with negative effects. This phenomenon is familiar: for example, although the objective of a negative list may have been to reduce drug costs, it could be that obliging physicians to prescribe other products will result in cost increases in unexpected areas (e.g., because of adverse reactions resulting in hospital admissions). The obvious solution would be to monitor more variables. However, if too many indicators are selected the evaluation may be rendered so complex or the measured outcomes may prove so contradictory that the findings are difficult to understand and act on. Finally, introducing too many indicators can render the entire process excessively costly and labour-intensive, the phenomenon that has been termed an “audit explosion”.
A further risk is that in selecting indicators there is sometimes a tendency to choose an indicator or target that is not relevant to a particular policy context or health care system. The indicators may not be relevant despite being simple to understand and may have been selected because they were likely to suggest a favourable outcome. This tendency to isolate clusters from their context is known in epidemiology as the Texas-sharpshooter fallacy, referring to the story of a Texas sharpshooter who shoots at the side of a barn and then draws a bull’s-eye around It is important not too adopt policies from abroad too rapidly merely because an audit carried out there has indicated that they are successful. Even if the foreign audit seems methodologically sound, the same policies implemented in one’s own country cause of differences in the medical and organisational context or other difficulties pointed out in earlier sections.
It is important too that any results arising from evaluations be disseminated and action taken so that necessary changes are made. Results should be disseminated to the appropriate decision-makers in a timely manner. Strong leadership and governance principles can be useful to ensure that appropriate changes are made in a uniform manner at a macro level and where applicable at local levels. Problems associated with the dissemination of evaluations are discussed in more depth in Chapter 5 of this book.
Before planning any policy evaluation one needs to consider carefully how the pharmaceutical service is structured, how it functions (i.e. its dynamics) and how the goals and objectives of intended policy measures (whether related to cost containment or not) could affect it. Policy priorities will have to be set in advance and means identified of assessing progress. Cost containment policies can themselves be costly as well as time-consuming, and only with good planning, the setting of clear priorities and careful assessment over time can one ensure that they are worthwhile and will do more good than harm.
The assessment of progress may well lead to the need to adjust, extend or even abandon particular policies. Since both the policies themselves, and any amendments to them as they are implemented, can have widespread implications for local structure and resources, health service managers should be involved in setting the agenda for the evaluations. In addition, the ultimate recipients of service - patients and professionals - should also be involved in assessing the effects of policies in the field. Most important of all, these ongoing measures of progress need to be studied, regularly and promptly, by the policy makers themselves, so that whatever policy changes are called for can be introduced without undue delay.
 L.A. Aday, C.E. Begley, D.R. Lairson and C.H. Slater, Evaluating the medical care system: effectiveness, efficiency and equity, Foundation of the American College of Healthcare Executives, Ann Arbor, Michigan, 1993.
 Audit Commission, A prescription for improvement: Towards more rational prescribing in general practice, HMSO, London, 1994.
 R.H. Brook and E.A. McGlynn, Maintaining quality of care, in: Health Services Research, E. Ginzberg, ed., Harvard University Press, Cambridge, 1991.
 P. Brudon-Jakobowictz, J.-D. Rainhorn and M.R. Reich, Model Indicators for Monitoring National Drug Policies, World Health Organization, Geneva, 1995.
 E.A. Chrischilles and K. Gondek, Do practice guidelines augment drug utilisation review?, Pharmacoeconomics 12(6) (1997), 648-666.
 A.J. Culyer and A. Wagstaff, Equity and equality in health and health care, Journal of Health Economics 12 (1993), 431-457.
 M.F. Drummond, Experimental versus observational data in the economic evaluation of pharmaceuticals, Medical Decision Making 18(2) (1998), Suppl. S12-S18.
 M.N.G. Dukes, Introduction, in: Drug Utilisation Studies - Methods and Uses, M.N.G. Dukes, ed., WHO Regional Office for Europe, Copenhagen, 1993.
 P. Dunleavy, So, what do political scientists do?, New Statesman 9 (9 October 1998), 3-5, 31.
 L. Greenfield, B.D. Colen, P.D. Cleary and S. Greenfield, Evaluating the Quality of Health Care: What Research Offers Decision Makers, Milbank Memorial Fund, New York, 1996.
 I.C. Gyssens and B.J. Kullberg, Improving the quality of antimicrobial drug use can result in cost containment, Pharmacy World & Science 17 (1995), 163-167.
 H. Hansluwka, Measuring the health of populations: Indicators and interpretations, Social Science and Medicine 20 (1985), 1207-1224.
 W.W. Holland, Evaluation of Health Care, Oxford University Press, Oxford, 1983.
 M. Johannesson, B. Joensson and G. Karlsson, Outcome measurement in economic evaluation, Health Economics 5(4) (1996), 279-296.
 D.M. Krumholz, S.D. Jaanus and R.L. Urquidi, Prescribing patterns of optometrists newly licensed in therapeutic drug use in New York State, Optometry 72(5) (2001), 292-298.
 J. Le Grand, Equity and Choice: An Essay in Economics and Applied Philosophy, HarperCollins Academic, London, 1991.
 C.R. MacIntyre, D. Sindhusake and G. Rubin, Modelling strategies for reducing pharmaceutical costs is hospital, International Journal of Quality in Health Care 13(1) (2001), 63-69.
 Management Science for Health, Managing Drug Supply, Kumarian Press, West Hartford, Connecticut, 1997.
 A. Mant, W.C. Rotem, L. Kehoe and K.I. Kaye, Compliance with guidelines for continuity of care in therapeutics from hospital to community, Medical Journal of Australia 174(6) (2001), 277-280.
 A. Maynard and T.A. Sheldon, Health economics: Has it fulfilled its potential?, in: Non-random Reflections on Health Services Research: On the 25th Anniversary of Archie Cochrane’s, A. Maynard and I. Chalmers, eds, The Nuffield Provincial Hospitals Trust, London, 1997.
 E.J. Mishan, Cost Benefit Analysis, 4th edn, Routledge, London, 1988.
 A.S. St Leger, H. Schnieden and J.P. Walsworth-Bell, Evaluating Health Services’, Open University Press, Milton Keynes, 1992.
 C.D. Shaw, Health-care league tables in the United Kingdom, Journal of Quality in Clinical Practice 17 (1997), 215-219.
 C.C. Veninga, P. Denig, L.G. Pont and F.M. Haaijer-Ruskamp, Comparison of indicators assessing the quality of drug prescribing for asthma, Health Services Research 36(1 Pt 1) (2001), 143-161.
 World Health Organisation, Quality Assurance in Health Care, Vol. 1, No. 2/3, WHO, Geneva, 1989.
 World Health Organisation/Action Programme on Essential Drugs, How to investigate drug use in health facilities: Selected drug use indicators, WHO/DAP, Geneva, 1993.