Starting or Strengthening a Drug Bulletin - A Practical Manual: 12. Evaluating quality and usefulness: 12.5 Assessing readers' opinions of the bulletin: 12.5.5 How many replies are enough?

Starting or Strengthening a Drug Bulletin - A Practical Manual
(2005; 165 pages)

Table of Contents

Preface
How the manual was produced
About ISDB
Executive summary
1. Introduction
	1.1 Are you starting or developing a bulletin?
	1.2 Objectives of this manual
	1.3 The need for the manual
	1.4 The importance of feedback
2. Rational use of medicines
	2.1 The relationship between evidence and rational use
	2.2 Other influences on the choice of medicines
	2.3 Sources of information for prescribers
	2.4 The special role of drug bulletins
	2.5 Specific ways in which bulletins can help
	2.6 Bulletins as part of wider initiatives for promoting rational use of medicines
	2.7 Summary
3. What are drug bulletins?
	3.1 Definition
	3.2 The history of drug bulletins
	3.3 What makes an ‘independent’ drug bulletin independent?
	3.4 Types of editorial content
	3.5 Styles of communicating information
	3.6 The institutional base
	3.7 References
4. Defining aims, target and type of bulletin
	4.1 Principles
	4.2 What is already available?
	4.3 Information on drug utilisation helps you choose topics
	4.4 Defining and refining the aims of the bulletin
	4.5 Who are the readers?
	4.6 What type of information is needed?
5. Planning resources
	5.1 The first steps in planning
	5.2 Identify what is already available or accessible
	5.3 Make a realistic assessment of additional needs
	5.4 Human resources: who will do the work?
		5.4.1 The editorial team
		5.4.2 The advisory board
		5.4.3 External reviewers
	5.5 Maintaining the motivation of contributors
	5.6 Material resources - an office, equipment and references
	5.7 Financial resources - the key to sustainability
	5.8 Long-term sustainability
	5.9 Cost-saving strategies
	5.10 Key messages for starting a drug bulletin
	5.11 References
6. Planning bulletin production: schedules and timing
	6.1 Why is planning necessary?
	6.2 Start modestly and grow gradually
	6.3 Develop a framework for producing articles
	6.4 Flexible planning for each issue
	6.5 Allow time for delays in distribution
	6.6 Integrate necessary sidelines into your overall planning
	6.7 A few principles for planning the production of a bulletin
7. The editorial process
	7.1 Outline of the editorial process
	7.2 Editing a drug bulletin
		7.2.1 Editorial independence
		7.2.2 Strong and sustained editorial policy and a committed team
		7.2.3 Selecting topics for articles
		7.2.4 Using and adapting existing material
		7.2.5 Planning a bulletin issue
	7.3 Writing bulletin articles
		7.3.1 Finding and motivating authors
		7.3.2 Rewarding authors
		7.3.3 Writing an article
		7.3.4 Outlining the topic
		7.3.5 Searching for documentation
		7.3.6 The first draft
		7.3.7 Discuss the draft with an editor
	7.4 Reviewing the article
		7.4.1 How many people should review an article?
		7.4.2 Should a bulletin set up a permanent review board?
		7.4.3 Should readers be included in the review board?
		7.4.4 Should patients or lay people be included?
		7.4.5 Should the pharmaceutical industry review drafts?
		7.4.6 Tips for reviewing articles
		7.4.7 Rewarding reviewers
	7.5. Rewriting the article
	7.6 Final checks
	7.7 Follow-up after publication
		7.7.1 Indexing
		7.7.2 Authors’ certificates
		7.7.3 Post-publication correspondence
		7.7.4 Corrections
		7.7.5 Requests for reproduction
		7.7.6 Keeping track of how bulletin articles are quoted and used
	7.8 References
8. Reviewing a new drug: is it a therapeutic advance?
	8.1 Introduction
	8.2 When is a new treatment a therapeutic advance?
	8.3 Collecting evidence about the drug
		8.3.1 Where to find evidence on new drugs?
		8.3.2 What about using unpublished information?
	8.4 Evaluation in terms of efficacy, harm and convenience
		8.4.1 Efficacy
		8.4.2 Adverse effects/harms
		8.4.3 Convenience
	8.5 Judging the overall value of the drug
		8.5.1 Considering the new drug in the local and individual context
		8.5.2 Overall rating scales
	8.6 Cost
		8.6.1 Compare costs in the light of true therapeutic value
		8.6.2 Comparisons should be appropriate and practical
		8.6.3 Bear in mind additional costs for using drugs
	8.7 What patients need to know
	8.8 References
Annexe to Chapter 8: Evaluating harm
	8.An-1 Talk about harm, not risk
	8.An-2 Assessment of causation of a harmful effect
	8.An-3 Assessing coherence with preclinical data
		8.An-3.1 General principles
		8.An-3.2 Chemical structure of new drugs
		8.An-3.3 Considering the profile of adverse effects
		8.An-3.4 Ratio of toxic and efficacy level in the same animal
		8.An-3.5 Extrapolation of animal toxicity (safety) level to humans
	8.An-4 References
9. Design and production
	9.1 Elements of good design
	9.2 Creating the design
	9.3 Using images
	9.4 The production process
	9.5 Developing a house style
	9.6 Ensuring accuracy - proof reading
	9.7 Printing
	9.8 Electronic publishing
	9.9 Further reading
10. Dissemination
	10.1 Why dissemination is important
	10.2 Managing a subscription-based approach
	10.3 Guidelines for effective distribution
	10.4 Communicating your bulletin’s messages more widely
	10.5 Key messages
11. Organizational and legal issues
	11.1 Introduction
	11.2 Why does a drug bulletin need a structure?
	11.3 Different kind of structures
		11.3.1 An association
		11.3.2 A foundation
		11.3.3 Other legal entities
		11.3.4 A bulletin within another organization
	11.4 How to deal with legal action
	11.5 Copyright
		11.5.1 Limits to copyright
		11.5.2 How is copyright created?
		11.5.3 How to use copyrighted material
		11.5.4 Fair use and quotes
	11.6 Further reading
12. Evaluating quality and usefulness
	12.1 Introduction
	12.2 Evaluation brings many benefits
	12.3 Three approaches: audit, feedback and impact assessment
	12.4 Start with your own evaluation of the bulletin
	12.5 Assessing readers' opinions of the bulletin
		12.5.1 Methods of assessing readers’ opinions
		12.5.2 Potential problems
		12.5.3 Selecting the data collection method
		12.5.4 Planning the survey
		12.5.5 How many replies are enough?
		12.5.6 Drafting the questions
	12.6 Evaluating the impact of the bulletin
	12.7 Feedback is achievable and invaluable
	12.8 Simple observations can tell a lot
	12.9 References
13. Partnership and collaboration
	13.1 The importance of supportive partners
	13.2 Possibilities at national, regional and international levels
	13.3 Various forms of collaboration
		13.3.1 Twinning arrangements
		13.3.2 Sharing information and resources
		13.3.3 Using established bulletins’ expertise
		13.3.4 Training sessions and workshops
		13.3.5 Ongoing information exchange between bulletins
		13.3.6 Collaborative research among bulletins
		13.3.7 Information and support for funding applications
	13.4 Identifying partners and networks
	13.5 Clearly define conditions for partnership
14. Keeping records and creating a memory
	14.1 Why keep records?
	14.2 What to keep and record?
	14.3 Creating an organizational memory
	14.4 How to start an archive
	14.5 Further reading
Appendix: Electronic sources of information
	For general access to evidence
	Journal articles
	Systematic reviews
	National disease prevention and control, health promotion etc.
	Health technology assessment
	Clinical guidelines
	Formularies/Essential medicines
	Regulatory authorities
	Adverse effects
	Information about/for patients
	Drug promotion
	Networks and directories
	ISDB member bulletins
	Feedback and evaluation form

12.5.5 How many replies are enough?

Sample size and selection of respondents need careful consideration. The simplest option is to send the survey to everyone on the mailing list. A response rate of no more than 20 - 25% is a realistic expectation. Where the mailing list is large, resources may not permit analysis of thousands of responses. Alternatively, you can select and send the survey to a representative sample of those on the mailing list.

In constructing a representative sample, you may want to consider demographic factors such as age, sex, individual or group practice, medical specialty, rural or urban location. This will help to make sure that you are reaching a range of different types of readers. The variables used depend on what you know about your readers. For example, if you know the location and specialty of those on the mailing list, select the sample by first grouping men and women separately, then subgroup by primary care doctor and specialist, then further subgroup by rural and urban location. If your mailing list is large, you can divide it into many subgroups. If it is small you may want to stick to larger groups of different types of readers. You can then randomly select names from each group. Most computerised statistical programmes include a random number generator. Consult a statistician on the number of individuals to be selected from each group.

Strategies for increasing the response rate include:

• A follow-up telephone call/letter to non-responders. Contacting non-responders is good if you know who they are. (This needs to be taken into consideration if the survey is confidential i.e. you have to be able to identify who has responded without knowing their name); or

• Replacing non-responders with another group of readers who meet the same randomisation criteria;

• Always assume you will get a poor response and send out more questionnaires than you think you need.

Box 12.2 Readership survey checklist

Purpose of the survey

• What are the bulletin’s aims? E.g. convey current drug information, advocate changes in regulation, etc.

• What readers are you targeting? E.g. all, or a subgroup such as primary care doctors, hospital doctors, pharmacists, the public.

• What sort of information do you need from this survey? e.g. extent bulletin is read, usefulness of the articles, suitability of electronic publishing.

• Whose opinion do you want? E.g. all on mailing list, or a particular subgroup.

• What will the survey achieve? E.g. ideas for improving the format, evidence to support continued funding.

Literature review

• Have similar surveys been done? Review the literature for published reports.

• Ask editors for survey reports and sample questionnaires via the ISDB network
[http://www.isdbweb.org].

• What errors were made and what problems were encountered in other similar studies? Talk to other editors and ask for a copy of their survey.

• Who could give advice? Ask an experienced editor or researcher.

Methodology

• Will you let participants know who the research is being conducted for? This has potential advantages and disadvantages.

• What questions need to be asked?

• How many readers should be sent the survey?

• How will they be selected?

• What demographic variables (profession, location, etc.) should be considered?

• Should respondents be anonymous? This is usually recommended to encourage a full and honest response.

• What is the likely response rate? Remember you may be working with busy medical professionals so this may not be as high as for other types of survey.

• Will non-responders be followed-up; if so, how will they be contacted?

• Is an inducement needed/appropriate to ensure a good response rate?

• Is it necessary to include a stamped self-addressed envelope?

• If the service is available, consider reply-paid envelopes - you only pay the postage for those returned.

• What is the cut-off date for processing responses?

• Is it sufficient for a few readers to review the questionnaire before full distribution or is a pilot study needed?

• How will the data be collected, processed and analysed?

Administration, workplan and budget

• Who has overall responsibility for the survey?

• Who will monitor activities/progress?

• Who will process and analyse the results?

• How long will it take? Is the estimated length of time realistic?

• What resources are needed (staff/money/data processing software etc.)?

• Do you have the appropriate analysis tool?

• Do you have the necessary expertise or access to expertise?

• Who could provide funding and equipment?

Data analysis

• Was the response rate adequate?

• Can valid conclusions be drawn from the results?

Publication and use of results

• Who needs to see the final report? Who is interested in your results? You may want to publish a summary of the results in the bulletin. This is a way of showing respect for respondents and valuing their contribution, as well as allowing readers to compare their attitudes to the overall trend.

• What actions are needed in response to the survey results? Who will do this?

• Who else could benefit from this research? You may want to submit an article on your survey and what you learned from it to the ISDB Newsletter.

• How could you have improved the survey?

• When should you next undertake an evaluation?

If you have divided people into subgroups using demographic variables, such as whether they are male or female, a primary care doctor or specialist, etc., include a question on these variables in the survey. This will help to show you whether a representative sample of people from each of these groups responded to your survey. Also, bear in mind that, in order to draw valid conclusions from the survey findings, you may need to be able to identify respondents (e.g. which responses are from general practitioners) so build this into the survey design.