11.3.1 Measures of regulatory effectiveness

How can the effectiveness of drug regulation be measured? Ensuring safety, efficacy and quality of drugs available to the population is the main objective of drug regulation. Effectiveness of drug regulation should, therefore, be judged according to the extent to which such objective is achieved by the drug regulatory functions. One way to measure drug regulation effectiveness is by relating the final outcomes of drug regulation to the various drug regulatory functions, and developing measures to capture the performance of regulation.

The ultimate measures of the effectiveness of drug regulation can be summarized as follows:

• drugs available are efficacious for stated diseases and conditions, and no dubious drugs are distributed on the market (the efficacy objective);

• drugs available are adequately safe (the safety objective);

• drugs available possess the stated quality characteristics (the quality objective);

• drugs are used in accordance with approved claims and methods (the rational-use objective).

The first three objectives relate to pharmaceutical products, and the fourth to pharmaceutical usage although, in most countries, promoting rational use of drugs is not part of regulatory activities. In order to achieve the four objectives, the various pharmaceutical activities-manufacturing, importation, exportation, distribution (wholesale and retail), production of product information and promotion and advertising-must be regulated by government. Measures to control these activities are embodied in several regulatory functions (see Figure 2.1):

• licensing and inspection of pharmaceutical establishments;
• product assessment and registration;
• monitoring of drug quality;
• controlling and monitoring promotion and advertising;
• ADR monitoring.

Each of these key drug regulatory functions is designed to accomplish the four interrelated objectives outlined above. Some regulatory functions relate directly to drug regulation objectives, while others play a complementary role in assuring successful implementation of another function. So in evaluating the effectiveness of drug regulatory policy, two concepts must be clear, namely:

• there is a multi-stage causal relationship between policy mechanisms and policy effects;

• policy mechanisms are essential for producing outcomes, but they are not enough by themselves.

Relationship between policy mechanisms and policy effects

Policy effects are the results of policy measures, but they are often affected by other issues or areas of activity in a multi-stage causal relationship. Therefore, drug regulation being a policy measure, implementing it may not lead directly to the desired final outcomes: other functions or activities may contribute, positively or negatively. Consider, for example, the policy of providing appropriate information on drug labels and information inserts to promote rational drug use (see Figure 11.2). This policy will be effective in promoting rational use if, and only if:

• the information contained on the label and insert is accurate and complies with regulatory requirements;

• the label and insert are included in the final drug packaging dispensed to the patient (and not discarded by the dispenser during any repackaging which occurs);

• the patient or carer can, and does, read the information;

• the information is clearly understood;

• the patient follows the instructions provided in the information package.

Failure at any one of these stages would break the link between the policy (the provision of information in the label and insert) and the desired outcome (rational use of the medication) (29). The items in bold type in Figure 11.2 indicate those preconditions which may be achieved by means of regulation.

This example shows not only that it is difficult to develop outcome measures, but that measuring outcomes may be rendered more difficult and complicated by other factors. On the one hand, spurious effects may be found, caused by factors other than regulatory functions. On the other hand, even when regulation does work, detection of the desired outcomes and their impact may be difficult.

Figure 11.2 Preconditions for rational drug use

Potential inadequacy of policy mechanisms for producing outcomes

Even though outcome measures may fail to detect the intended effects, this does not automatically mean that regulation is ineffective: it may be that this particular regulatory mechanism, although necessary, is unable to produce the intended outcomes alone. The challenge is to identify factors that will complement the regulatory mechanism in order to achieve the desired outcomes.

The following sections provide examples of how to measure regulatory effectiveness in four important areas of drug regulation (licensing and inspection of pharmaceutical establishments; product assessment and registration, including drug QC and ADR monitoring; control of promotion and advertising; enforcement of drug regulation).

As described above, drug regulation involves a number of interdependent functions and processes:

• licensing of manufacturing, importation and distribution of drugs, as well as of the parties involved in drug promotion and advertising;

• assessment and registration of products (marketing authorization);

• inspection and surveillance of pharmaceutical establishments;

• control of clinical trials;

• control of drug promotion and advertising;

• testing product quality (quality control);

• ADR monitoring.

For regulation to be effective, the necessary structures must be available and should include: enabling legislation and regulations; an organization to implement and enforce the legislation; adequate numbers of qualified staff; sustainable funds and other resources including physical resources (facilities, equipment, supplies, etc.); written guidelines, procedures and standards for implementation of the various functions.

11.3.2 Assessing the effectiveness of licensing and inspection

In almost all countries, drug laws require that establishments engaged in the manufacturing, importation, distribution or sale of pharmaceuticals should be licensed and regularly inspected.

Licensing, authorized by legal mandates, aims to establish appropriate settings for the manufacture, storage and distribution of drugs, and to ensure that manufacturing QC and dispensing activities are carried out under the supervision and guidance of qualified and approved personnel. The purpose of inspection is to confirm that licensing requirements are met and maintained throughout the processes of manufacturing, distribution and dispensing. The ultimate objective of both licensing and inspection is to ensure that drugs reaching the user and/or the consumer are safe and of sufficient quality for their intended use. Therefore, three main steps need to be monitored to ensure the effectiveness of regulation of pharmaceutical establishments:

• compliance with GMP requirements;
• compliance with good storage and distribution practices;
• compliance with dispensing practices.

The impact of licensing and inspection on the quality of products is also realized through multi-stage causal links, as shown below (Figure 11.3). Once again, those factors which can be influenced by regulation have been shown in bold type.

Figure 11.3 Relationship between good practice, quality of products and rational drug use

Consequently, measurement of the impact of licensing and inspection on the final outcomes (quality of product and quality of use) is complicated by other factors. The question then becomes: how can we measure the effectiveness of licensing and inspection? Two approaches are possible: a) checking whether the necessary structures for licensing and inspection activities exist or b) using indicators that provide information about the intermediate outcomes of licensing and inspection.

The existence of structures can be determined simply by asking questions that demand a “Yes” or “No” answer. For example, are there:

• legal provisions requiring the licensing of pharmaceutical establishments?

• written guidelines and procedures for the licensing of pharmaceutical establishments?

• written criteria and conditions which must be met to obtain a licence to operate pharmaceutical establishments?

• legal provisions requiring inspection of pharmaceutical establishments?

• legal provisions giving inspectors the power to enter and inspect pharmaceutical premises?

• guidelines on GMP?

• guidelines on GDP?

• checklists and procedures for inspection?

The effectiveness of the licensing and inspection systems can be assessed by using indicators which measure intermediate outcomes at the individual function level. For instance:

• number of licensed pharmaceutical establishments, compared with the total number which should be licensed;

• number of licensed pharmaceutical establishments, compared with the total number of pharmaceutical establishments operating in the country;

• number of licensed pharmaceutical establishments which were inspected before receiving license approval, compared with the total number of licensed drug establishments in the country;

• total number of inspections carried out, compared with the total number of inspections scheduled within a given period of time;

• number of licensed drug manufacturing plants that are currently GMP-compliant, compared with the total number of licensed manufacturing plants in the country.

As well as specifying conditions that pharmaceutical establishments must meet in order to obtain a licence, drug laws forbid unlicensed establishments or individuals to operate. In other words, the existence of pharmaceutical establishments that are not licensed may indicate that the licensing system is ineffective. The significance of knowing of the existence and the approximate number of such establishments is twofold: they are engaged in pharmaceutical business which may harm public health and they are not formal (licensed) pharmaceutical establishments, so they may not be subject to official regulatory activities and may therefore escape inspection. Information about such establishments is thus a necessary component in the assessment of all sources of pharmaceutical products in a country. Generally, information about the estimated number of unlicensed drug establishments is best obtained by conducting a field survey.

Examining the inspectors’ workloads will also help to establish reasons for the success or failure of the inspection process. During their inspection of pharmaceutical establishments, inspectors may learn of the presence of unregistered products, counterfeit products, etc. Quantitative information on these and other issues is valuable as an outcome measure.

Performance of the 10 countries

Global and regulatory spheres: The four major types of establishments regulated by the 10 countries are manufacturers, importers, wholesalers and retailers. Each of the countries regulates most or all of these drug sources, although there are some exceptions. For example, importers are not licensed in Cuba or Cyprus, owing to a lack of relevant legal provisions. The question, then, is whether those establishments operating without a licence perform in accordance with the standards necessary for ensuring quality products. In Cuba, the State is responsible for drug imports. Therefore, if the procedures for handling imports also ensure QC, they will compensate for the lack of legal provisions relating to licensing of importers. In Cyprus, although importers are not licensed, drug imports are subject to Pharmaceutical Services monitoring.

The study has shown that most of the countries have licensed their drug establishments in accordance with legislation. There are exceptions in Cuba, where only approximately 50% of the manufacturers are licensed (although all the unlicensed manufacturers are legal), and in Estonia, where only 4% of wholesalers are licensed.

However, it is unclear whether hospital pharmacies, clinics and health centres are regulated in the same manner as private-sector drug outlets.

The study indicates that DRA inspectors in six countries (Australia, Malaysia, the Netherlands, Uganda, Venezuela and Zimbabwe) have at some time detected illegal manufacturing, importing, selling and dispensing of drugs in their countries. Similar illegal activities are likely to exist in the other countries as well. Yet there are no official or reliable estimates of the number of establishments and people engaged in illegal activities related to drugs. The fact that illegal activities have been detected in six of the 10 countries suggests that their inspection systems are effective, since they clearly attempt to cover the informal sector as well.

Monitoring, violation and sanction spheres: Inspection activities are carried out in all countries to monitor the compliance of pharmaceutical establishments with legislation. As discussed in Chapter 7, most inspections in the majority of these countries are reported as being based on inspection plans. However, inspection plans, for both GMP and distribution channels, almost always refer to the formal sector, i.e. licensed establishments. This means that not all pharmaceutical establishments are controlled: illegal, unlicensed establishments are likely to be missed. They are a challenge to drug regulation because they are convenient channels for the distribution of smuggled, substandard and counterfeit drugs. They represent the gap between regulation and monitoring. Yet unless these illegal establishments are regulated, drug regulation cannot be considered effective. Therefore, in order to monitor all pharmaceutical establishments effectively, inspections should cover both licensed and unlicensed establishments.

Inspection plans provide varying degrees of coverage of manufacturers and distribution channels in the various countries. Two points are worth mentioning here. Firstly, the Australian TGA carries out a comparatively large number of GMP inspections per year. A system of targeted inspection is used, combining a risk-management approach with the “GMP history of manufacturers”. This enables human resources-the inspectors-to be directed to where inspection is more urgently needed, rather than inspecting all manufacturing establishments at random.

Secondly, structural features in drug regulation in countries operating a federal system of government affect the ability of the DRA to monitor drug distribution throughout the entire country. In Australia, authority over distribution channels is fully delegated to the individual states. As a result, the TGA does not have the authority to assess and control the drug distribution situation for the whole country. In contrast, the Pharmaceutical Services Division in Malaysia appoints a deputy director of health in each of its 13 states with power to issue licences, carry out inspections and submit reports. Under this arrangement, command and control may be exercised and an official channel established for information flow between the federal and state governments.

11.3.3 Assessing product registration

Product assessment and registration serve to ascertain the efficacy, safety and quality of drugs which are to be put on the market. However, the objectives of drug efficacy and safety are, at best, only partially met by the process of product assessment at the registration stage. Only limited research, including clinical trials, is conducted on a drug before it is submitted for registration review. Unknown effects, beneficial or adverse, will therefore probably manifest themselves at a later stage, after the drug has been registered and released for large-scale use. However, a system for monitoring ADRs makes possible the continuous assessment of a drug after it has been registered. Both functions are thus necessary to assure drug safety and efficacy.

Post-marketing surveillance of product quality is carried out by means of a QC system. The quality of products available on the market is therefore an important indication of the effectiveness (or otherwise) of the drug regulatory system.

Two key indicators of the effectiveness of registration are: whether all products on the market with a medicinal claim have been assessed and registered; and whether the efficacy, safety, and quality of products submitted for registration have been fully evaluated. Drug registration is not a bureaucratic formality, but a critical process.

Drug registration can be assessed by examining legal and administrative structures, as well as by measuring intermediate and final outcomes. Examples of indicators that provide qualitative and quantitative information on the effectiveness of drug assessment and registration are:

• existence of legal provisions specifying the types of products which must be assessed before they may be marketed;

• existence of written criteria for approval and registration of products;

• existence of written guidelines on how to assess products;

• existence of written policy on combination products;

• number of registered products, compared with the total number of products requiring registration in the country;

• categories of products by source (government, private, nongovernmental organization) that are currently being registered, compared with the total number of categories of products (by source) required to be registered;

• number of drug products found to be not registered, compared with the total number of samples of products collected and investigated.

However, comparing the number of products registered with the number of products requiring registration is not sufficient to assess the efficacy and safety of products claiming therapeutic effects. The percentage of drug applications rejected (out of the total number of drug applications received) and the number of drugs recalled because of adverse reactions (out of the total number of drugs registered) may also help to show how effectively the registration process functions.

Furthermore, in countries where combination products may not be registered, or where written criteria exist for considering such products, it is possible to measure the effectiveness of the drug assessment process by comparing the number of combination products which have been registered, but do not comply with the relevant guidelines, with the total number of combination products registered in the country.

Performance of the 10 countries

Global and regulatory spheres: Product registration is another area where a clear discrepancy exists between the global sphere and the regulatory sphere. Not all products that make therapeutic claims are subject to product assessment and registration in all countries. This means that not all drugs have been assessed before they are made available to the public. However, in some countries, the regulatory sphere is expanding. Australia, Malaysia and the Netherlands have recently instituted a system whereby herbal and homeopathic drugs must be registered. Such a system is also in place in Venezuela.

Australia, Cuba, Cyprus and the Netherlands register almost all drugs that are required by law to be registered. However, not all the countries manage to do the same. In Malaysia, for example, the percentage of registered drugs is low, largely because the process for registering herbal drugs is being introduced in stages and is not yet complete. In Uganda, greater efforts will have to be made if all drugs on the market are to be registered.

Monitoring, violation and sanction spheres: For post-marketing surveillance of product safety, ADR monitoring systems exist in all the countries, with the exception of Uganda. But it is difficult to evaluate the effectiveness of these systems in monitoring product safety. Only the Adverse Drug Reactions Advisory Committee in Australia receives a substantial number of ADR reports. For other countries, the numbers of reports are small and may not be sufficient to provide conclusive verification of adverse reactions. To overcome this constraint, the awareness and collaboration of health professionals in reporting should be increased, and information should be pooled by submitting reports to an international ADR centre, thereby increasing the number of ADR reports for evaluation.

Each of the countries operates a quality analysis system for post-marketing control of drug quality, albeit with vast differences in capacity. Data on the outcome measure for drug quality-the number of drug samples that failed quality tests compared with the total number of samples collected-are available in all the countries, except the Netherlands. Failure rates are high in some countries, e.g. Tunisia and Uganda. In Australia, high failure rates are found for herbal and other complementary products, compared with prescription drugs. Empirical data on sanctions applied in such instances are not available.

11.3.4 Assessing control of promotion and advertising

Promotion and advertising affect the way drugs are used. The ultimate aim of regulating drug promotion and advertising (information) is to promote the rational use of drugs. However, as described earlier, the link between drug information and drug use is not a direct one. Because of the nature of regulation, control mechanisms are applied only to ensure that information provided is accurate and not misleading. Assessing the effectiveness of regulation in relation to drug information therefore depends on evaluating whether, and to what extent, advertising and promotion materials and activities targeting consumers and health professionals comply with approved product information.

Performance of the 10 countries

Global and regulatory spheres: Legal provisions for controlling drug information exist in all the countries, with the exception of Cuba, where advertising and promotion of drugs are not permitted. Existing laws invariably apply to the traditional mass media, i.e. publications, radio, television and billboards. Newer information channels, such as the Internet, pose a challenge for information regulation.

Information on drug labels and inserts is controlled through the registration process in all countries. Pre-approval of promotional and advertising materials intended for the public are required in all countries except Estonia.

Monitoring, violation and sanction spheres: Systematic monitoring of drug information is not undertaken in all the countries: a number of countries rely on complaints filed either by competitor companies or by consumers. Self-regulation is also used in most countries, although there are no mechanisms for taking action against those who violate the regulations.

Empirical data which would enable the effectiveness of drug information control to be evaluated are generally unavailable. The lack of outcome data on information regulation is more serious than in other areas of drug regulation.

11.3.5 Assessing enforcement of drug regulation

Law enforcement is indispensable for regulation. Adequate sanctions help to deter future violations and have a significant impact on regulatory effectiveness.

Ideally, separate data for each regulatory function should be used for assessing law enforcement, since this will help to identify and clarify problems. However, the country data collected during this study are inadequate for this purpose. Moreover, where they exist, they are aggregate data for all drug regulatory functions. Data from three countries only are suitable for comparison. Figure 11.4 shows the relative weight of violations and sanctions in terms of overall areas of work of the drug regulatory authorities in Australia, Venezuela and Zimbabwe. The numbers used for this are four-year averages for the period 1994-97.

Figure 11.4 Drug regulatory violations and administrative and judicial sanctions imposed, 1994-97

The sanction rates in Australia appear low compared with those for Venezuela and Zimbabwe. Nevertheless, the absolute number of both violations and sanctions are higher in Australia. A higher violation figure may be due to more severe violations, more extensive monitoring, or better-targeted monitoring. In Australia, targeted and extensive monitoring have been employed for products suspected to be of inadequate quality. This is an indirect indication of the effectiveness of the TGA in protecting the public from exposure to hazardous drugs. In Venezuela and Zimbabwe, administrative measures are used much more extensively than judicial measures-indeed, almost exclusively. For 1994-97 judicial measures were used in 6.3-10.7% of cases in Venezuela, and less than 1.6% of cases in Zimbabwe. In Australia, the percentage was much higher- 6.5-31.9%.