Starting or Strengthening a Drug Bulletin - A Practical Manual
(2005; 165 pages) View the PDF document
Table of Contents
View the documentPreface
View the documentHow the manual was produced
View the documentAbout ISDB
View the documentExecutive summary
Open this folder and view contents1. Introduction
Open this folder and view contents2. Rational use of medicines
Open this folder and view contents3. What are drug bulletins?
Open this folder and view contents4. Defining aims, target and type of bulletin
Open this folder and view contents5. Planning resources
Open this folder and view contents6. Planning bulletin production: schedules and timing
Open this folder and view contents7. The editorial process
Open this folder and view contents8. Reviewing a new drug: is it a therapeutic advance?
Open this folder and view contentsAnnexe to Chapter 8: Evaluating harm
Open this folder and view contents9. Design and production
Open this folder and view contents10. Dissemination
Open this folder and view contents11. Organizational and legal issues
Close this folder12. Evaluating quality and usefulness
View the document12.1 Introduction
View the document12.2 Evaluation brings many benefits
View the document12.3 Three approaches: audit, feedback and impact assessment
View the document12.4 Start with your own evaluation of the bulletin
Open this folder and view contents12.5 Assessing readers' opinions of the bulletin
View the document12.6 Evaluating the impact of the bulletin
View the document12.7 Feedback is achievable and invaluable
View the document12.8 Simple observations can tell a lot
View the document12.9 References
Open this folder and view contents13. Partnership and collaboration
Open this folder and view contents14. Keeping records and creating a memory
Open this folder and view contentsAppendix: Electronic sources of information
 

12.6 Evaluating the impact of the bulletin

Once you have conducted some assessments and gained confidence, and your bulletin is well established, with production running smoothly and a fairly stable readership, you may want to look for opportunities to evaluate the impact of the bulletin. Impact evaluations are usually undertaken to find out what effect a particular bulletin article has had on its audience, i.e. whether a behavioural change actually occurred following publication of the article, and to establish a cause-effect relationship. All drug bulletins strive for long-term change in the culture and discipline of prescribing, dispensing, and explaining drug use, and thereby improving patients' health. Even a small change in prescribing behaviour could have a big effect if the bulletin has a large readership or if the drug is prescribed frequently. For example if a drug bulletin in a developed country achieved a 0.5% improvement in the prescribing of statins it would benefit many people. A carefully designed impact study is needed if you want to know whether your bulletin is having a direct effect on the behaviour of readers. Following the publication of an article you may wish to assess readers' actual:

• prescribing of a particular medicine;
• communication with patients on a specific topic; or
• knowledge about a particular therapy.


Although it is desirable to assess whether the bulletin has caused readers to change their behaviour, this requires a high level of research skill and is time-consuming and costly. It is, therefore, useful to build up experience and skill on simpler evaluations before attempting an impact evaluation. Crucially, it depends on the required data being available. For example, few regions or countries have databases of prescribing statistics. Even if they are available, the data may not be reliable or they may not be able to reveal the trends of interest. When using data that have been collected for another purpose, you cannot be sure that the bulletin caused the change that you observed, as other influences cannot be excluded. For example, if an unnecessarily hazardous drug is used less often than previously, it may be because of your bulletin article, it may be because doctors received similar information from other sources, or it may be for other reasons, such as lack of availability, higher price, etc.

Box 12.4 Can a bulletin change practice on its own?

There is a fair bit of controversy in the scientific literature over whether providing health professionals with written information on its own is enough to bring about changes in prescribing. A US study found that articles can significantly change behaviour.4 However, a Cochrane review of 11 studies concluded that the impact is at best small and of uncertain significance,5 while a Dutch study found the effectiveness of information in changing behaviour was variable.6

More recently a Canadian study looked at this question with particular regard to a series of regular and expected printed educational bulletins, Therapeutics Letter.7 The study sought to measure the impact on prescribing to newly treated patients. A paired, cluster randomised community design was used. The study population included 499 physicians and 12 issues of the bulletin, Therapeutics Letter were followed-up. Physicians in the control group (n = 241) received the letters 3-8 months after physicians in the intervention group (n = 258). The impact on prescribing to newly treated patients (defined as patients who had not previously made a claim for any medication from the class of drugs profiled in the letter) was analysed. The probability of prescribing a drug recommended in the Therapeutics Letter rather than another drug in the same class increased by 30% in the three months after the mailing of the letter relative to the preceding three months, adjusted for any before-after changes in the control group (relative risk 1.30; 95% confidence interval 1.13-1.52). No letter achieved statistical significance on its own. However, 11 of the 12 letters produced prescribing changes in the predicted direction such that the overall result was significant when their effect was combined. The researchers concluded that the combined effect of an ongoing series of printed letters distributed from a credible and trusted source can have a clinically significant effect on prescribing to newly treated patients.

Types of impact studies include:

• Testing reader knowledge on a topic before and after the publication of an article.

• Reviewing prescriptions for a particular medicine or therapeutic group. This could be used to see whether a newly recommended dosing regimen has been adopted by prescribers.

• Simulated patient surveys, where an ‘actor’ consults a number of doctors and acts out a set scenario on each occasion. The doctor’s response is carefully recorded by the simulated patient(s). Each doctor is consulted by a simulated patient before and after publication of a particular article and the pre- and post-publication responses compared to see whether there has been a change in actual behaviour; however, there are potential practical and ethical problems with this kind of research.


When designing an impact study consider:

• Potential confounders - factors unrelated to your bulletin that can also cause the same result. For example, an editorial published during the study period potentially distorted the outcome of the Dutch impact study.6 Choose issues where it is possible to eliminate some confounders and control the impact of others on the study results.

• The length of the study. An impact study should be repeated after some time to see whether the behavioural change was sustained.

• Use of controls. Controls are essential to enhance the detection of differences and confounders. In the British Columbian Ministry of Health’s study to ascertain the influence of educational interventions and adverse media publicity on the prescribing rate for calcium-channel blockers,8 inadequate controls limited the ability to infer causation. A large control group will be needed if you are trying to detect a small behavioural change. Ask a statistician for advice on the size of control and study groups.


Large robust impact studies are unlikely to be achievable for all bulletins. More targeted impact studies can, however, provide valuable information about a particular aspect of a bulletin. For example, the independent evaluation of the impact of the readers' test of la revue Prescrire showed physicians who used the test were more knowledgeable than those who did not.9

to previous section
to next section
 
 
The WHO Essential Medicines and Health Products Information Portal was designed and is maintained by Human Info NGO. Last updated: March 20, 2014