Starting or Strengthening a Drug Bulletin - A Practical Manual
(2005; 165 pages) View the PDF document
Table of Contents
View the documentPreface
View the documentHow the manual was produced
View the documentAbout ISDB
View the documentExecutive summary
Open this folder and view contents1. Introduction
Open this folder and view contents2. Rational use of medicines
Open this folder and view contents3. What are drug bulletins?
Open this folder and view contents4. Defining aims, target and type of bulletin
Open this folder and view contents5. Planning resources
Open this folder and view contents6. Planning bulletin production: schedules and timing
Open this folder and view contents7. The editorial process
Open this folder and view contents8. Reviewing a new drug: is it a therapeutic advance?
Open this folder and view contentsAnnexe to Chapter 8: Evaluating harm
Open this folder and view contents9. Design and production
Open this folder and view contents10. Dissemination
Open this folder and view contents11. Organizational and legal issues
Close this folder12. Evaluating quality and usefulness
View the document12.1 Introduction
View the document12.2 Evaluation brings many benefits
View the document12.3 Three approaches: audit, feedback and impact assessment
View the document12.4 Start with your own evaluation of the bulletin
Close this folder12.5 Assessing readers' opinions of the bulletin
View the document12.5.1 Methods of assessing readers’ opinions
View the document12.5.2 Potential problems
View the document12.5.3 Selecting the data collection method
View the document12.5.4 Planning the survey
View the document12.5.5 How many replies are enough?
View the document12.5.6 Drafting the questions
View the document12.6 Evaluating the impact of the bulletin
View the document12.7 Feedback is achievable and invaluable
View the document12.8 Simple observations can tell a lot
View the document12.9 References
Open this folder and view contents13. Partnership and collaboration
Open this folder and view contents14. Keeping records and creating a memory
Open this folder and view contentsAppendix: Electronic sources of information

12.5.6 Drafting the questions

Once planning is complete, develop the questions. Consider what questions to ask, how they will be asked, and the order (see Box 12.3).

Box 12.3 General rules for drafting survey questions2

1. Avoid asking two questions at once e.g. 'Do you consider articles are accurate and unbiased?' These two points need to be divided into two questions.

2. Word questions so that all respondents understand the same meaning.

3. Be as precise as possible in the framing of questions e.g. 'Have you recently changed the advice you gave a patient because of an article?' would be clearer if worded as 'In the last six months, have you changed the advice you gave to a patient because of an article?'

4. Tell respondents when a question can have more than one response, e.g. 'You can tick more than one box'.

5. Design the survey to make the task of reading questions, following instructions and recording answers as easy as possible.

6. Word questions so that clear, unambiguous answers are more likely to be given.

7. Simple tick/check boxes work well for answer options, along with providing space for respondents to write additional comments.

8. Consider using questions with similar scales to one another so that respondents can become familiar with the task they are undertaking.

9. If using agree/disagree statements, try to ensure a mix of positive and negative statements.

10. Remember to allow for all possibilities in your responses: this may include having a ‘don’t know’ or ‘none of these’ for some questions.

The questions asked will depend on the information being sought. Here are some ideas from which to develop questions:

• The extent to which the bulletin is read.

• What happens to each issue?

• Factors preventing more articles being read.

• General readability of the bulletin.

• Preferred authors (national/international, specialists/primary care doctors etc.).

• Level of detail required in articles.

• Usefulness of various articles or sections.

• Content of articles (e.g. should clinical advice be given? If it is already, is the level of advice adequate?).

• Perceived influence of the articles (on prescribing practice, advice given to patients etc.).

• Acceptability of format, suggestions for improving design.

• Frequency of bulletin editions.

• Comparison with other journals.

• Suggestions for future articles.

Many questions merely require a yes or no response, or a comment. Those that measure degrees of frequency, satisfaction or other such parameters require presentation of a scale from which the respondent can select.3 Appropriate options have to be provided. Where there may be interpretation problems, consider using scales respondents are familiar with. It is always helpful to design your survey form with enough space for people to enter comments about the bulletin, which could be unrelated to the questions you asked them.

Measuring frequency

One approach is to ask the respondent to estimate the frequency of a certain action e.g. 'In the last six months, how many times did you refer back to previously published articles?' However, respondents may not remember how often they did something so providing a response scale with a rough quantification may be useful e.g. 'How often do you refer back to a previously published article?'

Every few weeks
Every few months
Less often than every six months

Avoid using a response scale of ‘often’, ‘sometimes’, ‘occasionally’, ‘rarely’ etc. as they are difficult to interpret.

Asking evaluative questions

Questions asking respondents to say how they rate specific aspects of a bulletin are common in readership surveys, e.g. what do you think of the standard of the book reviews? A numerical scale from 0 to 5, where 5 is excellent and 0 is poor, could be used or a scale such as:

Very good

To encourage accurate and full completion of the survey, keep scales simple (i.e. restrict the number of different answer options) and ensure there is a clear distinction between scale levels. If the differences are too subtle, this will affect the quality and accuracy of the survey results.

Measuring feelings

The following scale is often used to rate how respondents feel about something:

Very positive
Generally positive
Mixed: about equally positive and negative
Generally negative
Very negative

A numerical scale can also be used to measure feeling, for example, 'Do you feel the bulletin improves your ability to recognise an adverse reaction in a patient?' (on a scale of -2 to +2, with 0 being the midpoint). Note: while every point on the continuum does not need to be defined, there should be a midpoint where positive feelings turn to negative.

Rating agreement

Often readership surveys ask how much the respondent agrees with certain statements e.g. 'Some readers have suggested including a summary at the beginning of each adverse reaction article. How do you rate this suggestion?' An appropriate scale may be:

Strongly agree
Strongly disagree

While it may be appealing to offer a middle category (neither agree or disagree) it may be more useful to make the respondents commit themselves.

Ranking the bulletin

If, for example, you want respondents to prioritise your bulletin against three others publications for usefulness, list the publications and ask the respondent to rank them from 1 to 4, where 1 is the publication they find most useful, and 4 is the publication they find least useful.

Data processing and analysis

While data from simple readership surveys can be processed manually, a software package may be needed to process more complex survey responses. There are many data processing software packages available - ask a local researcher for advice about which to use.

Unlike clinical studies or full impact evaluations, statistical analysis of readership surveys is not usually necessary. If statistical analysis is required, ask a statistician for advice when planning the research.

At the end of this chapter there is an example of a readership survey questionnaire together with the results, from the French bulletin Bulletin d’Information du Médicament et de Pharmacovigilance.

to previous section
to next section
The WHO Essential Medicines and Health Products Information Portal was designed and is maintained by Human Info NGO. Last updated: March 20, 2014