Once you have conducted some assessments and gained confidence, and your bulletin is well established, with production running smoothly and a fairly stable readership, you may want to look for opportunities to evaluate the impact of the bulletin. Impact evaluations are usually undertaken to find out what effect a particular bulletin article has had on its audience, i.e. whether a behavioural change actually occurred following publication of the article, and to establish a cause-effect relationship. All drug bulletins strive for long-term change in the culture and discipline of prescribing, dispensing, and explaining drug use, and thereby improving patients' health. Even a small change in prescribing behaviour could have a big effect if the bulletin has a large readership or if the drug is prescribed frequently. For example if a drug bulletin in a developed country achieved a 0.5% improvement in the prescribing of statins it would benefit many people. A carefully designed impact study is needed if you want to know whether your bulletin is having a direct effect on the behaviour of readers. Following the publication of an article you may wish to assess readers' actual:
• prescribing of a particular medicine;
• communication with patients on a specific topic; or
• knowledge about a particular therapy.
Although it is desirable to assess whether the bulletin has caused readers to change their behaviour, this requires a high level of research skill and is time-consuming and costly. It is, therefore, useful to build up experience and skill on simpler evaluations before attempting an impact evaluation. Crucially, it depends on the required data being available. For example, few regions or countries have databases of prescribing statistics. Even if they are available, the data may not be reliable or they may not be able to reveal the trends of interest. When using data that have been collected for another purpose, you cannot be sure that the bulletin caused the change that you observed, as other influences cannot be excluded. For example, if an unnecessarily hazardous drug is used less often than previously, it may be because of your bulletin article, it may be because doctors received similar information from other sources, or it may be for other reasons, such as lack of availability, higher price, etc.
Box 12.4 Can a bulletin change practice on its own?
|
There is a fair bit of controversy in the scientific literature over whether providing health professionals with written information on its own is enough to bring about changes in prescribing. A US study found that articles can significantly change behaviour.4 However, a Cochrane review of 11 studies concluded that the impact is at best small and of uncertain significance,5 while a Dutch study found the effectiveness of information in changing behaviour was variable.6
More recently a Canadian study looked at this question with particular regard to a series of regular and expected printed educational bulletins, Therapeutics Letter.7 The study sought to measure the impact on prescribing to newly treated patients. A paired, cluster randomised community design was used. The study population included 499 physicians and 12 issues of the bulletin, Therapeutics Letter were followed-up. Physicians in the control group (n = 241) received the letters 3-8 months after physicians in the intervention group (n = 258). The impact on prescribing to newly treated patients (defined as patients who had not previously made a claim for any medication from the class of drugs profiled in the letter) was analysed. The probability of prescribing a drug recommended in the Therapeutics Letter rather than another drug in the same class increased by 30% in the three months after the mailing of the letter relative to the preceding three months, adjusted for any before-after changes in the control group (relative risk 1.30; 95% confidence interval 1.13-1.52). No letter achieved statistical significance on its own. However, 11 of the 12 letters produced prescribing changes in the predicted direction such that the overall result was significant when their effect was combined. The researchers concluded that the combined effect of an ongoing series of printed letters distributed from a credible and trusted source can have a clinically significant effect on prescribing to newly treated patients.
|
Types of impact studies include:
• Testing reader knowledge on a topic before and after the publication of an article.
• Reviewing prescriptions for a particular medicine or therapeutic group. This could be used to see whether a newly recommended dosing regimen has been adopted by prescribers.
• Simulated patient surveys, where an ‘actor’ consults a number of doctors and acts out a set scenario on each occasion. The doctor’s response is carefully recorded by the simulated patient(s). Each doctor is consulted by a simulated patient before and after publication of a particular article and the pre- and post-publication responses compared to see whether there has been a change in actual behaviour; however, there are potential practical and ethical problems with this kind of research.
When designing an impact study consider:
• Potential confounders - factors unrelated to your bulletin that can also cause the same result. For example, an editorial published during the study period potentially distorted the outcome of the Dutch impact study.6 Choose issues where it is possible to eliminate some confounders and control the impact of others on the study results.
• The length of the study. An impact study should be repeated after some time to see whether the behavioural change was sustained.
• Use of controls. Controls are essential to enhance the detection of differences and confounders. In the British Columbian Ministry of Health’s study to ascertain the influence of educational interventions and adverse media publicity on the prescribing rate for calcium-channel blockers,8 inadequate controls limited the ability to infer causation. A large control group will be needed if you are trying to detect a small behavioural change. Ask a statistician for advice on the size of control and study groups.
Large robust impact studies are unlikely to be achievable for all bulletins. More targeted impact studies can, however, provide valuable information about a particular aspect of a bulletin. For example, the independent evaluation of the impact of the readers' test of la revue
Prescrire showed physicians who used the test were more knowledgeable than those who did not.9
Loading...
