This section of the questionnaire aimed to i) determine the degree to which impact had been assessed and the methodology used, and ii) gather information that would help guide other project planners towards success and avoidance of problems. These are discussed under impact evaluation and process evaluation.

Availability of evaluation reports and general findings

Of those projects reported as completed, just over half have conducted a formal evaluation, and from18% to 27% of those still ongoing have done so. However, only 11 developed and two developing country projects sent a copy of their evaluation; this strongly reflects difficulties encountered in communication and public education work to obtain written reports from national and local projects.

Evaluations varied in objectives. Few attempted to measure impact. Most looked at the reach and progress of project activities. The following are examples of evaluations as reported by responding projects.

• The Swedish project, Young People and Medicines, carried out a careful evaluation that focused on the implementation of the project activities (how much education on prescription drugs was carried out at various levels of the school system). It did not look at acquired knowledge or new behaviours on the part of the ultimate target audience.

• An “evaluation” sent by a consumer NGO in Bolivia is actually a progress report.

• A journal article about a project in Peru (Study-intervention on morbidity and use of drugs) describes the study in considerable detail but does not identify impact on behaviour.

• A Canadian project, Wise Use of Medicines Campaign, gives a qualitative description of apparent behaviour changes on the part of elderly individuals, but also states that "the extent to which (these behaviour changes have) occurred requires further measurement".

• Australia's Be Wise with Medicines Month, 1992, sent a very comprehensive review and evaluation report. This campaign consisted of several of the projects listed as individual respondents to the present survey, including the SHAPE campaign, the Medicines Information Project (MIP) and the Pills and Older Persons Project (POPP). The evaluation encompassed the progress, reach and behavioural impact of activities carried out. Interestingly, although implementation was excellent and reach was relatively high, the "Omnibus Survey" showed a moderate level of awareness acquired (consistent with other public awareness campaigns) but no effect on current behaviours.

• The Hai Phong Health Financing Project in Viet Nam provided another example of a comprehensive evaluation. Although this project is multi-faceted, a separate evaluation was conducted of its Information-Education-Communication (IEC) component. This project communicated four very specific messages through a variety of channels. The evaluation consisted of household interviews, in-depth interviews, and focus group discussions. It attempted to assess exposure to messages, the relative effectiveness of the different channels used, the quality of the messages, and changes in behaviour. The project time was too short (4 months) to effectively change behaviour. However, the evaluation was able to measure the first three indicators. The results for using health workers and mass-media (mainly radio and commune loudspeakers) as channels for RUD information in rural Viet Nam are encouraging.

Impact Evaluation

Despite the low percentage of projects having carried out evaluations, about two-thirds of developing country projects and three-quarters of developed countries responded to the request for an estimate of project results. Two-thirds of all responding projects offered suggestions concerning factors facilitating success, constraining factors, and problems and lessons learned during implementation.

Of those respondents, 40% from developing countries judged that their project had met its objectives, compared to 66% of those from developed countries. 60% and 30% said the objectives had been partially met. No developing country and only 4% of developed country respondents felt that the project had not been successful. These estimates were chiefly determined through informal assessments and surveys. In developing countries these were primarily household surveys, whereas developed countries had a variety of survey types, including surveys carried out in schools and pharmacies. A few projects observed sales patterns of particular drugs or categories of medicines. Some additional methods of assessment included counting the number of telephone calls to the information line, tallying requests for project materials, and measuring process indicators (e.g. implemented activities).

Criteria for evaluation

Developing country projects based their evaluations on proportions of reported changes in knowledge (54%) and behaviour (43%). However, few respondents actually sent evaluations that included behaviour change. Developed country projects used more frequently the criterion of exposure to messages and materials (40%). This is consistent with the finding that developing country projects were more likely to use household surveys. There is a fairly even distribution across certain other criteria, such as change in consumer satisfaction, or change in sales patterns.

Figure 11 - Criteria used to determine impact

Additional criteria used in particular cases include changes in legislation, feedback from participating pharmacists, and audience response. For example, legislation was mentioned by the "Safety in Medicines" campaign in the USA and by the Consumer Education project in Malaysia. Feedback from participating pharmacists was measured by the French project on patient information cards and by a project discouraging black-market medicines in Côte d'Ivoire. Several different versions of audience response were reported, including applause to the Buko street theatre performances in Germany, feedback from students in Mauritius, and requests for information about DES in Canada.

Coverage

Coverage of project activities seemed to be difficult for most respondents to estimate. This is shown by the fact that approximately half of all respondents ticked "don't know", and that only 41% of developing and 20% of developed countries listed a coverage figure or estimate. This lack of coverage information also makes it difficult, if not impossible, to estimate the cost-effectiveness of a given project or intervention.

Of those projects able to measure coverage, responses covered a wide range, for example "15% of pharmacists" (France); "about 50% of NGOs" (India); "70% of parents" (Indonesia); "40% of pharmacists" (Australia); "91% of public schools [in the state where the project was implemented] participated" (USA). One project reported having exceeded the expected target. Other projects listed numbers of people contacted, classes held, or participating pharmacies.

Coverage estimates should also be compared with other indicators, such as the number of printed materials developed and distributed. For example, a primary health care programme in Bolivia reported that the coverage was "national; many reached". However, only 10 000 of the twice-yearly leaflets, 5 000 copies of comics for children, and 1 000 annual copies each of two posters were produced. A project in Mexico reached 3 000 community pharmacies with 3 000 copies of a calendar; however, only 300 copies of a poster for the general public were distributed. Other projects, reporting to be national in scope, produced only 5 000 copies of posters (Malawi), or 90 000 copies of a newsletter (USA). In projects of large geographic and social scope, these are inadequate quantities to have any national or exposure.

Process Evaluation

Problems in implementation

Problems encountered by projects in implementing activities are shown in Figure 12. These relate to lack of adequate and timely funding, control/coordination, opposition by the pharmaceutical industry, personnel/time, and motivation to change. However, the order of importance varied greatly between developing and developed country projects. Perhaps unsurprisingly, the most common implementation-related problem reported by developing countries was a lack of adequate and timely funding. Developed country projects listed funding problems in second place, but had more difficulties with sufficient personnel and time to carry out planned activities.

Developing country projects experienced problems of coordination and control, and what was described as "interference from the drug industry". These categories were not important for developed country projects (only one developed country mentioned problems with the drug industry). An important proportion of developing country projects reported problems concerning motivation to change on the part of the target audience. This is expressed as "difficult to change pharmacists' behaviour"; "behaviour change is slow”; "knowledge is not practiced"; and "resistance by doctors". A few developing country projects mentioned such problems as an inappropriate project design, or having too great a response and being unable to manage it.

Figure 12 - Problems experienced in implementing projects

Constraints to success

Constraints are defined as any factors which inhibit or limit the implementation of public education activities. Respondents were requested to report both external and internal constraints to project success.

• Internal constraints

The implementation-related problems were reflected fairly consistently in the listings of internal and external constraints to success. For developing and developed countries alike, funding/resources was listed as the first most common internal constraint. Time is a very important internal limiting factor for developed country projects. A third constraint, for both developing and developed countries, is lack of internal coordination, or the presence of internal discord and disagreement.

• External constraints

The most frequently-mentioned and important external constraint was lack of external collaboration and support. This category includes such responses as the "lack of recognition by powerful professional groups"; "slow response from Ministry of Health (MOH)"; the lack of inclusion of the topic in formal school curricula; bureaucracy; and "GPs were not involved". These comments represent about one-third of all external constraints, mentioned by developing and developed countries alike.

Inadequate funding/resources represents the third most important external constraint for developing countries, and the second most important for developed country projects.

The lack of external collaboration and support is closely related to two other external constraints, carrying equal importance for developing and developed country projects: competing forces and unsupportive legislation. The former includes incentives given by other organizations (for instance to get health workers to focus on other projects); drug company marketing or lobbying, and "organized opposition to the project". Examples of legislation-related constraints include "lack of appropriate legislation regarding drugs"; "ill-defined legislation"; and change in regulation. (Unfortunately these last comments remain opaque, as the respondents did not elaborate on their statements.)

Facilitating factors

Factors contributing to success, or facilitating factors, are defined as any factors which stimulate, provide, or promote a fertile environment for public education. For this survey, external factors which contribute to success fell into the categories of support by other organizations, support by the media, and supportive legislation, while internal factors related more to funding, strong planning, stable personnel, and internal cooperation.

By an overwhelming margin, the most frequently-listed category of external "success" factor for both developing and developed country projects is support by other organizations. This includes support from the ministry of health, from various health services, from a narcotics control board, from collaborating institutions such as a literacy group or community groups, and from professional groups such as pharmacists or medical and allied health professionals.

Eight developed country projects, from Australia, Sweden and the USA, and one developing country project (Mexico) presented an apparent contradiction. While they found that having collaboration and support from some source was a facilitating factor, these projects also felt constrained by the lack of collaboration and support from a different source. For example, one project said "MOH support" was a facilitating factor, but that "having government partners would have helped". Another project said they had support from a literacy group and the Plain English campaign, but not from their MOH; a third had support of the MOH for the research component of their project but ran into apathy on the part of medical school authorities. This indicates that even projects with external support do not always perceive it as adequate.

With the exception of funding, most of the constraints and problems discussed earlier were further supported in the lists of internal and external factors that contributed to success. Other external factors listed repeatedly include supportive legislation, and support by the media.

Internal cooperation had a significant influence on the success of a project. However, the most important facilitating factors were reported to be competent, sufficient, and stable personnel to plan and carry out the work, and strong planning of the project. Other responses of consequence included having distinct objectives, a clear understanding of the problem, and committed staff and volunteers. It is worth emphasizing that the factors contributing to success seemed to have the same importance whether the project was carried out in a developing or a developed country. This will be important when formulating recommendations and guidelines for future projects.

Lessons learned

Respondents listed up to three lessons learned during the planning and implementation of the projects., shown in Table 2. Not surprisingly, these lessons reflect once again the problems encountered, the constraints, and the factors contributing to success.

Table 2 Lessons learned

The most consistently-mentioned category of lessons learned, for both developed and developing country projects, is the need for better collaboration with professional partners. Some expressions of this are: "must work with professional associations"; "contact all partners in advance"; "must have... prescriber support"; "outside educators should be consulted"; "need participation of all sectors involved in drug use". This is consistent with the most frequently-encountered project constraint.

Another frequently-mentioned category of lesson, for both developing and developed countries, deals with the need for better planning and clear objectives. This planning includes a clear knowledge of the health problem, as well as the need to allocate sufficient time and personnel for research. Other lessons learned include a need for better collaboration with the target group, and the recognition that behaviour change is difficult and long-range.

It is significant that not all lessons learned were based on negative experience. On the contrary, the second most frequent category of lessons learned for developing countries and the third most common for developed is that public education on rational drug use can work. A primary health care project in Bolivia says, "It is possible to get the community to take responsibility for their health". The campaign on RUD in Bangladesh says, "Continuous, logical insistence to producers on rational/ethical production WORKS", and "(these) activities are a good example of effective lobbying". The Safety of Medicines campaign in the USA learned that "Commercial, consumer, and professional interests can be brought together", and the Medicine Information Project in Australia says, "Empowering consumers can drive change at all levels of the health system".

Mirroring these reports of success is a small but important category of lessons learned: there is a need for education on RUD. This is expressed by projects in Cameroon, Chile, Colombia, India, Malaysia, Sierra Leone; Australia, Germany, New Zealand, Switzerland, and the USA.