Ms Elodie Jambert
Department of Essential Drugs and Medicines Policy, World Health Organization, Geneva

Introduction

The Internet can be a valuable resource for users seeking health information. By its very nature, the Internet was designed to provide a forum for the unrestricted exchange of information, and web sites may contain factual information, opinions, data, ideas, propaganda, self-promotion, and/or commercial material [8]. However, the quality of health information on the Internet is extremely variable and difficult to assess. Unlike traditional print resources, web resources rarely have editors or fact-checkers, and this tends to be even more so for national web pages [9]: the sites are not monitored, edited, regulated or approved. There are no rules or standards governing the type or quality of information that a writer, or an organization, can put on the Internet, and there is no consensus on how to resolve the problem, no uniform guidelines for quality assessment of web-based health information for consumers. Because this could potentially affect health outcomes for millions of people, there is a growing urgency for objective, reproducible, widely accepted criteria that can be used both to evaluate the quality of information on a web site [10] and to prepare a reliable, good-quality, easy-to-use web site.

For most patients, health professionals, policymakers, and pharmaceutical companies, the information available on the web site of a drug regulatory authority (DRA) should be the most appropriate way to obtain information about drugs. If used properly, the Internet allows DRAs to make information about their activities available to other authorities and organizations, as well as to the public and industry. By having a reliable, easy-to-use, accessible, and up-to-date web site, a DRA can increase its efficiency and efficacy, while at the same time being recognised as being more transparent in its regulatory work. While health information proliferates on the Internet, and e-trade increases worldwide, it is crucial for health professionals and patients to have access to reliable information, particularly the list of authorized products in their country. This should also include all laws and regulations related to medicinal products. Such transparency helps to maintain public confidence in pharmaceutical products.

Background

There have been concerns raised by the national authorities of several WHO Member States regarding the issue of cross-border trade of pharmaceuticals through the Internet. In 1999, WHO published a booklet called Medical products and the Internet: A guide to finding reliable information [3]. In October 2000, a workshop took place in Geneva, on ‘Pharmaceuticals and the Internet’. Some key aspects that were discussed included: What should be done in addition to the existing WHO guidelines? How should illegal e-trade be fought? How should the safety issues be addressed? etc.

The recommendations included countering biased information and malpractice, making more reliable information readily available, and analysing the situation as regards DRA web sites. It was suggested that a WHO Model Web Site for DRAs should be created to ensure the availability of good-quality, reliable information.

In order to assist WHO Member States, WHO has taken the initiative in providing criteria and guidance for existing or new DRA web sites, addressing the following concerns:

• uneven quality of pharmaceutical information available on the Internet;
• e-trade - unregulated and uncontrolled dispensing and self-prescription;
• misleading and biased advice on pharmaceuticals, dietary supplements, etc.

Objectives of the study

The purpose is to propose a WHO Model Web Site as a guide for DRAs to create their own web site or improve it, in order to offer Internet users improved quality and increased availability of health information on the Internet. Before the project could begin, the most important pieces of information that a DRA web site should contain were identified in a list of objective, reproducible criteria.

An assessment was then carried out to provide a brief overview of the quality of the information currently on existing DRA web sites - by analysing their content - in different languages and regions and at different levels of sophistication.

By making pharmaceutical information available on reliable, accessible web sites, DRAs will greatly speed up the work of health professionals involved in drug procurement and supply. Better prescribing by medical doctors will also be facilitated and rational use of drugs improved. Just as important, communication and a transparent dialogue among DRAs, industry, consumers/patients, and health professionals will be enhanced.

More generally, DRAs will

• increase the transparency of their own activities;
• improve collaboration between DRAs, health professionals, and academics;
• improve the public-health impact of drug regulatory work;
• facilitate networking between DRAs to solve drug regulation problems;
• offer reliable and unbiased drug information to guide Internet users;
• help combat the problem of substandard and counterfeit drugs by offering reliable information.

Methodology

The development of a list of criteria designed to serve as the ‘gold standard’ to make the analysis was the essential first step. A list of the countries that have a DRA web site was compiled either by searching the web or contacting the ministries of health and DRAs.

In parallel, bibliographic research was undertaken to find relevant articles on how to assess a health-information web site. Many organizations [11] and individuals have published criteria to evaluate health-related information on the Internet. A literature and Internet search found that ‘the most frequently-cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors or developers, currency of information (includes frequency of update, freshness, maintenance of site), authority of source, and ease of use, and accessibility and availability’ [12].

These criteria are very general and do not fully correspond to the evaluation of DRA web sites, which needs more ‘specific’ criteria. A set of key criteria (general and specific) was developed on different types of drug information (lists of approved drugs; national drug regulations; information on how to ensure safe, efficacious, and rational use of specific drugs; lists of approved companies and their authorized activities, etc.). Twenty-four criteria were identified (six general and 18 specific) and set up as a checklist to assess the quality of information on a DRA web site.

A scoring system ranging from 0 to 2 (0 = inadequate, 1 = intermediate, 2 = good) was used to weight each criterion. To use this scoring system, each criterion was reviewed through content (quantitative aspect), links (selection, content, etc.), interactivity (mechanism for feedback, forum, etc.), accessibility, and balance of information for targeted users (industry, health professional, public).

General criteria

1. User friendliness

The first impression when looking at a web page and its general appearance [13] are very important.

• Most pages are designed attractively and to entice further exploration.

• Information is presented logically and clearly enough to be successfully manipulated by the intended user and is easy to find.

• Unless the national language is English, French, or Spanish, information is both in the national language and in one of the three widely used languages.

• Spelling and grammar are correct.

2. Navigability

This permits a site to be used effectively and enables the user to get to the important information [14]. Accessibility, logical organization, and internal search engines are essential.

• Users can get the information they need within a reasonable number of clicks (preferably three or fewer) [15].

• Users can move from page to page, link to link, and item to item with ease, without getting lost or confused.

• Users do not need to pay a fee or type in personal information (such as name or e-mail address) before using the site.

• If there are large amounts of information on the site, some kind of search function should be provided.

3. Speed

The home page and most subsequent links (except those from web sites outside the control of the DRA) are displayed in up to 4 to 5 seconds.

4. Site map

• A site map shows logical lines and organization of the site.
• It shows clearly how to navigate through all pages.
• Users can easily find out where to go and how to get there. The ‘buttons’ or menu items are labelled using clear language [16].

5. Search

The site has its own search engine, set up for searching either by key words or with an ‘A-Z’ index. It also permits searches for other sites.

6. Update

The web site includes the date on which it was created and the date of the last update. The site is regularly updated and reviewed. This criterion is very important; if the information has not been updated since the previous year, it is not very reliable.

Specific criteria

1. Mission statement

The mission statement and purposes of the DRAs are clearly stated and easily accessible. Activities and programmes are described.

2. Contact information

Contact persons with names, addresses, phone and fax numbers, and email are readily available for each activity, complete and up to date. It is easy to identify person/service and to reach information.

3. Organizational structure

The structure of the DRA is available, complete, and up to date. It allows information on a department to be obtained by clicking on it.

4. Services

• The site offers information related to the DRA.

• The information is clearly labelled and organized, and easy for users to understand.

• The site does not contain words that try to ‘persuade’ rather than ‘inform’.

5. News, events, and meetings

The site mentions and describes news, events, and meetings planned and has a calendar that is transparent and regularly updated.

6. Safety alerts and adverse drug reactions (pharmacovigilance)

• Safety alerts - patient and health professionals have easy access to the safety alerts by date and product name: recalls, suspensions, revocations, and recall procedures. The information is comprehensive, detailed, and chronological. Batch size is also mentioned.

• Adverse drag reactions (ADRs) - the site contains a definition of terms, procedures for reporting adverse drug reactions and adverse drug events, how to report ADRs on-line, links to useful sites, last pharmacovigilance reports, etc.

7. Feedback form for informing the DRA

There should be a facility enabling any user to contact the authority, either to ask questions or to report problems. This feedback form should be easy to download, complete, find, and send back to the DRA.

8. Regulatory guidance on legislation and regulations

Regulatory guidance such as laws, decrees, orders, and any legislative and regulatory material related to pharmacy, drug manufacturing, drag registration processes, commerce, information, promotion and advertising, and e-trade should be available.

9. Instructions for applicants

• Information for applicants should be presented with a sufficient degree of detail to prepare registration of dossiers: composition of the file, fees, document to file for first demand, variations, renewal, extension, transfer of marketing authorization, the drag registration process and procedures, accelerated registration procedures, and norms, standards and guidelines for pharmaceuticals.

• Forms (that can be downloaded) should include a marketing authorization application form, a variation application form, and a template for a summary of product characteristics.

• Information on drag regulation and quality-assurance systems, drug regulatory information, guidelines for good clinical practice for trials on pharmaceutical products, etc., should be included.

10. Medicinal products (human/veterinary medicines)

This criterion is very important. The web site should include the following information, which should be in more than one language and updated monthly.

• the list of authorized drugs in the country, including at least

• generic name;
• authorized presentations (form and strength);
• patient information leaflets;
• summary of product characteristics;

• a search facility that permits the user to find items either by brand name, holder of marketing authorization, INN, therapeutic group, or status of marketing authorization;

• the list of cancelled marketing authorizations;

• information about orphan drugs;

• products under special post-marketing surveillance monitoring.

11. Approved manufacturers

Information or statistics on manufacturers in the country should be presented and easily readable: a list of approved manufacturers (name, addresses, contacts), licence status, last inspection date, etc. If charts or graphics are included, their quality should be good and should enhance the content rather than distract from it [17].

12. Import and export

Statistics on import and export; guidance for importers, exporters; WHO type certificate, etc., should be presented and easily readable.

13. Approved wholesalers, distributors, pharmacies

A list of approved wholesalers, distributors, and pharmacies should be provided, with information on their activities and volume of business.

14. Basic statistics on drug consumption

The site should offer information about sales data, manufacturing data, etc.

15. Basic statistics on country profile

The site should contain basic statistics about the country, such as the following:

• population data;
• population income statistics;
• health statistics;
• healthcare facilities, etc.

16. Basic statistics on DRA activities

There should be information on how many new drug applications the DRA has received; the number of pending applications; number of positive or negative decisions; number of applications withdrawn; inspections of pharmacies, wholesalers, and manufacturing sites.

17. Links

The links provided to other pages and sites should meet the following criteria:

• They should include links to sites that are worthwhile and appropriate for the intended audience.

• They should be clearly labelled and serve an easily identified purpose.

• They should give added value to the web site.

• They should be current/unexpired and should operate efficiently.

• They should be grouped in some type of logical order.

Links to the web sites of medical journals and national drug regulatory authorities should be included, along with international links (World Health Organization, European Agency for the Evaluation of Medicinal products, etc.).

18. Publications

• Sources of information cited should be reliable, pertinent, and identifiable.
• Publications can be downloaded with a current bibliography of references.

The publication page can contain, for example, the DRA’s bulletin, annual report, quarterly report, cumulative list of recalls, safety alerts (and other decisions that restrict use of medicinal products), guidance materials, latest list of approved products, latest list of approved manufacturers, wholesalers, importers, distributors, medical journals, newsletters, and periodicals, etc.

Results

Only 53 national DRA web sites were found; 51 were evaluated (the sites of Austria and Greece were not assessed in this study). Five of the 53 sites are only available in the national language (Austria, Greece, Italy, China, Portugal).

Broken down by WHO regions, the DRA web sites are distributed as follows (given as percent of countries in the region): EURO (55.8%), WPRO (32.1%), AMRO (19.5%), AFRO (8.5%), SEARO (30%), and EMRO (4.5%) (see Annex 2). Details of the analysis of the DRA web sites is shown in Table 1.

Findings

• Only 13.7% of the web sites scored ‘good’ on information on medicinal products.

• Only 11.8% scored ‘good’ on safety information.

• Basic statistics: only 7.8% scored ‘good’ on drug consumption information; 5.9% on wholesalers, distributors and pharmacies; 5.9% in manufacturers; 7.8% in import and export in the country; 11.8% on country profile; and 21.6% on DRA activities.

• In 56.9% of the web sites, no feedback form was provided for informing or contacting the authority.

• There was no recent update (during the current year) indicated on 54.9%.

• There was no text search in 56.9%, or the search was not functioning.

• No publications at all were mentioned by 58.8%.

• The links function scored ‘inadequate’ on 51% of the web sites assessed. Also, ‘services’ were rated ‘inadequate’ on 71.4%, although this score is very subjective.

• Only 27.5% had a good site map; 60.8% had no site map at all.

• There were good pages on news and events in 27.5%; 52.9% had nothing related to such topics.

• The best results were found for speed (56.9%), navigability (58.8%), contact information (37.3%), and organizational structure (41.2%).

• The results for ‘instructions for applicants’ were ‘good’ on 47.1% of the sites.

• The results were ‘intermediate’ for mission statement (51%), user friendliness (49%), and regulatory guidance on legislation (39.2%).

Table 1: Percentages of scores of ‘inadequate’, ‘intermediate’, and ‘good’ for the 51 DRA web sites

Discussion

Only a few DRAs across the globe provide their respective information resources to an Internet audience via a web site. Indeed, many DRAs do not have a web site, and even those that have one seem to provide either a small amount of information or such a vast amount of information that it is difficult to navigate through the site. Most web sites do not post approved drug information, and very few provide access to the list of authorized drugs in the country. The quality of health information provided is very heterogeneous and could benefit from improvement.

Lacking drug information, healthcare providers and consumers may be uncertain as to which drugs are approved for treating which conditions, which drugs meet national regulations, and which drugs may be imported. For consumers, these problems are being exacerbated by the growth in pharmaceutical e-trade. Ensuring the rational use of drugs and an efficient drug supply then becomes more difficult. At the same time, lack of drug information can suggest that DRAs are not ‘transparent’, ultimately leading to a lack of trust in the quality and legitimacy of their work.

Conclusions

DRAs should provide reliable drug information for consumers/patients, health professionals, and public- and private-sector organizations (hospitals, manufacturers, importers, etc.), making it more widely available and easily accessible. And the quality of the information provided has to be improved.

The Internet is a powerful information tool that could be used for exactly this purpose. Indeed, it has proved that it can be a source of valuable, good-quality information on the approved drugs, treatments, and medical products available in different national markets. For many DRAs, this will mean creating a web site. For others, the task will be to improve an existing web site.