Ideally, the hospital will have a drug information centre to handle requests concerning adding drugs to the formulary or requesting changes to the STG. If not, a pharmacist or a physician can provide the necessary drug evaluations, given the time and at least some of the resources listed above. However, very few pharmacists or physicians take the time or have the skills to accurately evaluate a journal article describing a drug study. Health professionals frequently read the abstract and conclusions with little or no attention to the structure and validity of the written article. They may therefore fail to recognize articles based on poorly designed studies with inaccurate or invalid conclusions. National DTCs and tertiary hospitals must review the primary literature, i.e. the actual drug studies.

However, for most hospital or sub-national DTCs, review of good-quality secondary or tertiary literature should be sufficient. It is not necessary for different centres to all review the same literature, nor do most of them have the time and capacity to do so.

Discussion of critical review of the primary literature is beyond the scope of this manual. However, it is important that DTC members have some skills in this type of critical review in order to better assess and use commonly available secondary and tertiary literary sources and literature from the pharmaceutical industry. For literature concerning a new medicine to be sufficient for a DTC to decide whether to add or delete a medicine from the formulary, it should:

• Compare the drug of interest to another standard drug in its class and not just to a placebo or another drug of poor performance. Unfortunately in many studies a new drug is compared only to a placebo or to a drug of poor performance.

• Test the drug of interest in patients that are representative of those who would take the drug in the DTC’s institution and not just in healthier ‘study patients’. Whether the sample of patients is representative and relevant can only be judged by a description of the inclusion and exclusion criteria for patients in the study.

• Measure clinically important outcomes, for example blood pressure or blood sugar, using established methods, for example relative or absolute risk reduction (see section 4.4); the amount by which a clinical outcome is improved (for example mmHg for blood pressure) is just as important as whether the difference between one medicine and another is statistically significant.

• Use adequate study design, preferably a randomized controlled trial (see section 7.6), and test the medicines in a sufficient number of patients; this is necessary in order to ensure that any observed effects are not due to factors (confounders) other than the medicine being tested and are also not due to chance. Trials comparing a drug against a placebo will require at least 40 patients to demonstrate symptom relief and usually several thousand patients to demonstrate a reduction in mortality. Several hundred to several thousand patients are required to show superiority of one drug over another.

• Take adequate precautions to ensure that the results are not biased. If possible, patients, prescribers and any researchers judging clinical outcomes, should be blinded to which medicine a patient is taking; this will ensure that their opinions do not influence the results (measurement bias). Patients should be randomly selected to receive the new drug, comparator drug or placebo and the random selection process concealed from patients and professionals alike; this will ensure that there are no differences that could influence the results between patients receiving the new drug and those receiving the comparative drug or placebo (selection bias).

• Apply appropriate statistical analysis to the results.

- p values of less than 0.05 are taken by convention to mean that the results of a study are not due to chance. A p value of 0.05 indicates that there is a 1 in 20 probability that any study result is due to chance, meaning that there is a 5% chance of observing a result which does not exist in the population. This means that there is a 95% chance that any difference observed, for example, between the drug of interest and the comparator drug, is a true difference in the population.

- The power of a study indicates the likelihood of a hypothesized result being observed and is dependent on sample size. A value of 80% is taken by convention to be the minimum and indicates that there is an 80% chance of observing a real difference, for example, between the drug of interest and the comparator drug, meaning that there is a 20% chance of not observing a difference that really exists in the population.

- The confidence interval indicates the range within which the true study results lie. By convention 95% confidence intervals are used and indicate that there is a 95% chance that the true result lies within the estimated or observed range. The larger the sample size the narrower the confidence interval of an observed value (for example, mean reduction in blood pressure or percentage of patients with pain relief).

• State its funding sources and whether it has been peer reviewed; this is necessary because studies funded by the pharmaceutical industry are often only published if they are positive and in journals that are peer reviewed less strictly or not at all.

More detail about common problems seen in many drug studies is summarized in annex 4.2, with a checklist to use when critically reviewing articles.