- Medicine Information and Evidence for Policy > Monitoring and Evaluation
- Traditional Medicine > Traditional, Complementary and Herbal Medicine
(2000; 80 pages) [French] [Spanish]
3.3 Selection of Study Design
Clinical research aimed at evaluating traditional medicine should incorporate the conventional concepts of research design, such as randomized controlled trials or other types of clinical studies, such as observational studies. The USA Food and Drug Administration guidelines Guidance for industry: significant scientific agreement in the review of health claims to conventional foods and dietary supplements, which introduce several types of clinical studies, could be consulted (see Annex VII). The Guideline for Good Clinical Practice produced by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (see Annex VIII), as well as official guidelines from other governmental agencies (such as those in Annex V) may also be a good reference source for clinical research design.
Conventional concepts of clinical research design may be difficult to apply when using clinical research to evaluate various systems and practices of traditional medicine, depending on the goal of the assessment (see Part 3, section 3.1). In such circumstances, the choice of study design should be discussed on a case-by-case basis with experienced traditional medical practitioners. The study design may be chosen from a whole spectrum of clinical research designs which are suitable for assessing traditional medicine (see Annex VII), including:
Single-case designs have the advantage of being adaptable to the clinical needs of the patient and the therapeutic approach of the practitioner, but have limitations due to their lack of generalization to other patients. Such designs are appropriate for the development of research hypotheses, testing those hypotheses in daily clinical practice and refining clinical techniques. Single-case designs using a common protocol-if the protocol can be systematically followed-should be advocated for collaborative research among practitioners from different backgrounds. For example, single-case designs can evaluate the effectiveness of various specialized acupuncture methods in patients with a variety of individual differences. In a single-case design, the patient is his or her own control. Treatment can be randomized for a patient, rather than the patient being randomized for a treatment.
The study of traditional medicine can also be undertaken in a “black-box” manner. This means that the treatment and all of its components are delivered as they would be in the usual clinical situation. In this type of study, no component of the treatment “package” is isolated and studied independently. This allows the effectiveness of traditional medicine to be determined either within its own theoretical framework or within that of conventional medicine.
Ethnographic studies that document the social and cultural context in which a traditional practice emanates may be appropriate in situations where there is no available scientific literature or other documentation. These and other qualitative studies can provide baseline information from which hypotheses may be generated, and can lead to further research.
Observational studies collect findings on a therapeutic or prophylactic treatment under routine conditions. The special feature of these studies is that they seek, as far as possible, not to influence the individual doctor-patient relationship with respect to indications, and the selection of and carrying out the treatment. These studies may be conducted with or without a control group. The specific details of the study (e.g. the time and extent of examination for each individual patient, the number of patients involved) and the envisaged methods (e.g. data recording and evaluation) must be adapted to the question investigated in the study (e.g. safety or appropriate posology). Observational studies have specific advantages in studying aspects of clinical safety. The use of such studies to prove efficacy is limited because bias in patient selection may occur. Nevertheless, the level of evidence on efficacy of traditional medicine can be significantly increased by well-designed observational studies.
Study outcome measures
It is essential that the outcome measures chosen be appropriate to the research question. Appropriate outcomes may include quantitative and qualitative outcomes; primary and/or secondary outcomes; and generic and/or highly specific outcomes.
Selection of patients
It is essential that the sample represent the target population of patients to which the results would be generalized. Publication of the study requires a clear description of the patients using both traditional and conventional terms. The reliability of the categorization/diagnostic criteria used in the study should be considered and stated. The source of the patients under study should be comprehensively described along with details of the recruitment process. The inclusion and exclusion criteria should be completely described and rationalized. Any potential bias in patient selection, recruitment and enrolment should be excluded. Investigators should be aware of any potential errors that may occur when studying traditional medicine out of context and without reference to its traditional theories and concepts.
When the research involves techniques that depend on skills that may differ between practitioners, such research should be conducted by more than one practitioner in order to increase the generalizability of the results.
The number of patients in a study needs to be adequate, in order to be able to determine any clinically important differences between the study groups. With respect to the study design, the statistical methods used should be appropriate to the proposed analysis of the study’s outcome.
A well-conducted and controlled clinical trial could provide sufficient evidence to establish a relationship between the use of a herbal medicine or traditional procedure-based therapy and the prevention, diagnosis, improvement or treatment of an illness, provided there is a supporting body of evidence from observational or mechanistic studies.
Randomized controlled trials require one or more control groups for purposes of comparison. The selection of control groups depends on the objectives of the study. In the evaluation of traditional medicine, a concurrent control group should be used. The control groups may involve (not in order of priority):
• well established treatment
• different doses of the same treatment
• sham or placebo treatment
• full-scale treatment
• minimal treatment
• alternative treatment.
Different controls can be used in clinical trials to answer different questions. The use of a placebo, when possible, is desirable, because it generates evidence of better quality. Placebo-controlled trials are intended to establish whether treatment is valuable over and above what might be achieved by a control treatment, and not whether treatment is valuable at all. Thus, it allows researchers to distinguish specific from non-specific effects of treatment in order to determine whether the additional cost, risk and effort of a specific treatment are worthwhile. It is also important for understanding the mechanism of a treatment. This is true for the evaluation of all drugs. It is not only of academic interest, but is also of practical value, especially for developing new treatments from traditional ones. However, in some cases, placebo-controlled trials may not be possible (see section on clinical trials in Part 1).
It is preferable to compare a herbal medicine with both a well-established treatment and another control group (from the list of control groups) to determine whether the herbal medicine is useful in the context of current best practice.
One specific problem in clinical research of traditional medicine is the simultaneous conventional treatment of patients (e.g. cancer patients) in a study. It may not be ethically possible to withdraw the conventional treatment. Therefore, in such cases, the focus of research may be on the additional or supportive effects of traditional medicine. Research on combinations of traditional and conventional medicine should always consider potential therapeutic interactions and side-effects (see section on black-box design in Part 3).
Randomization has been a tremendous advance in developing comparable groups to assess therapeutic interventions. It is essential to control various known, and even unknown, biases. Nevertheless, there are many situations where randomization can be impossible or unethical. The best way to solve this problem is probably by the proper selection of control treatments.
Blind assessment is a critical component of conventional evaluation of therapeutic interventions. However, in the evaluation of efficacy of traditional procedure-based therapies (such as physical therapy, surgery, acupuncture and manual therapy), it can be difficult, impractical or impossible for the practitioner to be kept ignorant of what treatment the patients are receiving. It is essential that this be noted in the evaluation of the validity of a study and that the judgement on its validity be applied consistently across all systems of conventional and traditional medicine.
Treatment blinding in the evaluation of herbal medicines should adopt the approach of conventional medicines, e.g. using active and control formulations with similar colour, taste and weight. However, if the herbal medicine cannot be administered in a predetermined standardized formulation, it will be impossible to keep the treatment blinded. Treatment blinding is also difficult to implement in most types of traditional procedure-based therapies. It is important, however, to reduce any bias introduced by non-blinded treatment by carrying out a blinded assessment of the primary outcomes of the study.
Evaluation of quality of life
Traditional medicine is used not only to prevent, diagnose, improve and treat illness, but also to maintain health and improve the quality of life. For example, traditional medicine may not cure patients with certain illnesses, such as cancer and AIDS, but may help improve their quality of life. The WHO QOL user manual, developed by the WHO Programme on Mental Health, can be used to help evaluate the results of clinical research on herbal medicines and traditional procedure-based therapies (see Annex IX)
Other issues related to therapeutic interventions
In both the development of a study protocol to assess traditional medicine and in its submission for publication or for health-authority approval, the following information regarding study outcomes should be clearly provided:
• description of the therapeutic intervention;
• description of the reasons for the selection of the therapeutic intervention;
• description of the rationale for the choice of the study outcomes;
• description of the outcome measurements, including a review of the validity and reliability of the measurements;
• a comprehensive protocol for taking the measurements (including how and when the measurements were taken); and
• a clear statement of which expected outcomes the statistical method was based on.
The following issues should also be considered:
• The type of intervention must be clearly defined. In treatment using herbal medicines, this should also include, for example, information on the composition and manufacturing of finished herbal products. In traditional procedure-based therapy, this should include, for example, information on the tools and equipment used.
• The training, skills and experience of the traditional medical practitioner should be taken into account. Issues concerning the variability of treatment by a single practitioner (intra-practitioner variability) and groups of practitioners (inter-practitioner variability) should be addressed. Ideally, the practitioner’s diagnostic ability should be reliable.
• If the setting is an important component of a treatment, its essential features must be described.
• The dose, frequency and duration of a treatment must be described completely. “Dose” in traditional procedure-based therapies refers to a variety of attributes related to each episode of the therapy, which may vary markedly between different systems of traditional medicine. In acupuncture, for example, “dose” includes the force of a physical manipulation, duration of each episode of therapy, duration of needled manipulation, the number of repetitions of a procedure, the number of needles used, the depth of stimulation, the needle sensation if elicited, the details of any electrical stimulation including stimulus, frequency, intensity, etc. The “dose” used in any study should be based on the relevant literature and experience of traditional medical practitioners.
• The duration of follow-up should be clearly stated. Its length needs to be appropriate to the treatment carried out. In patients with acute pain, follow-up should be carried out within a 24-hour period. In patients with chronic pain, follow-up of a minimum of several months (e.g. 3-6 months) is desirable.
• Temporal considerations need to be assessed and noted. The study design should take into account seasonal variations that are important to some traditional medicine systems. It should also contain an appropriate time course to allow the treatment to demonstrate its effectiveness. The number of treatments in a finite period of time needs to be clearly stated.
The information in the ICH Harmonized tripartite guideline: guideline for Good Clinical Practice issued by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use is a useful example of the information required (see Annex VIII).