Subnavigation

• RHL home
• Adolescent sexual and reproductive health
• Fertility regulation
• Gynaecology, infertility and cancers
• HIV
• Improving clinical practice
• Newborn health
• Pregnancy and childbirth
• Sexually transmitted infections

Audit and feedback: effects on professional practice and health-care outcomes

RHL Commentary by Pattinson RC

1. EVIDENCE SUMMARY

This review (1) sought to assess the effects of 'audit and feedback' on the practice of health-care providers and patient outcomes. The specific objectives of the review were: (i) to ascertain if audit and feedback is effective in improving professional practice and health-care outcomes; (ii) how the effectiveness of audit and feedback compares with that of other interventions; and (iii) whether audit and feedback can be made more effective by modifying the process of its implementation.

The authors used a broad definition of audit and feedback: “Any summary (written or verbal) of clinical performance of health care over a specified period of time.” In the studies included in the review, the type of feedback was often not specified and was thought by authors of included studies to be implicit in the audit process. Overall, the authors of the review found variable results in the studies included in the review and, when effective, the effects of audit and feedback on practice were generally small to moderate.

For each comparison made in the review, the review authors calculated the risk difference (RD) and the risk ratio (RR). To explain the variation in the effectiveness of interventions across comparisons, the authors investigated the following factors: type of intervention (audit and feedback alone, audit and feedback with educational meetings, or multifaceted interventions that included audit and feedback), intensity of the audit and feedback, complexity of the targeted behaviour, seriousness of the outcome, baseline compliance, and study quality.

The intensity of audit and feedback was categorized in terms of the recipient (an individual or a group), format of feedback (verbal and/or written), and the source, frequency, duration and content of the feedback. Audit and feedback categorized as “intensive” was attributed the following characteristics: individual recipient, receiving from a supervisor or senior colleague, over a moderate to prolonged duration. Complexity of the targeted behaviour was graded as being “high,” “moderate” or “low” depending on the number of behaviours requiring change, nature of judgments/skills needed to implement change, and whether the change required change in an individual or the organization. The seriousness of outcome was assessed as high, moderate or low, with acute problems with serious consequences considered high, primary prevention considered as moderate and numbers of tests or prescriptions considered as low.

A total of 118 studies are included in the review. For sharply contrasting (dichotomous) outcomes, the adjusted risk difference (RD) of changing over to the desired practice ranged from –0.16 (a 16% absolute decrease in compliance) to 0.70 (a 70% increase in compliance); the adjusted risk ratio (RR) varied from 0.71 to 18.3. For continuous outcomes (outcomes measured on a scale that is continuously variable), the adjusted percentage change compared with controls varied from –0.10 (a 10% absolute decrease in compliance) to 0.68 (a 68% increase in compliance) (median = 0.16, inter-quartile range = 0.05–0.37).

The authors found that the effect of audit and feedback was more likely to be greater when at baseline health professionals' adherence to recommended practice was low, and when audit and feedback was provided more “intensively” with or without educational meetings. There was no other effect that could be detected, but that did not mean that other effects were absent. The quality of available studies was often poor. The review authors further postulated that the effects of audit and feedback might be larger if health professionals were actively involved in, and had specific and formal responsibilities for, implementing change. There was no information on the costs of performing audit and feedback.

The search for, and retrieval of, trials was extensive. The extraction and analysis of the data was a complex exercise owing to the diverse range of health professionals among which trials were undertaken. The authors have thought carefully about how to classify the studies and extract the information. The data presentation is not the usual forest plot usually associated with the Cochrane Reviews, but box plots and adjusted risk ratios and adjusted risk differences. Careful reading is required to understand this review, but the effort is well worthwhile.

2. RELEVANCE TO UNDER-RESOURCED SETTINGS

2.1. Magnitude of the problem

From a common sense perspective it is easy to appreciate that audit and feedback would be a useful way to re-examine existing hospital protocols and replace the ones found to be ineffective with new protocols that have been shown to be effective. “Morbidity and mortality review meetings” are an ideal starting place for such reviews. Unfortunately in smaller health institutions in many developing countries mortality and morbidity review meetings are not held routinely. For example, in South Africa, the holding of maternal and perinatal mortality meetings (although encouraged) is largely restricted to larger hospitals and academic institutions. A necessary first step in introducing change would be to establish these meetings and make them part of the culture of each health-care institution, big or small. Once established there would be a platform from which to analysis problems and develop appropriate solutions. These meeting would also be the first steps towards establishing a system of clinical governance in all health-care institutions. Review of performance of health-care professionals should be an integral part of all health-care professionals’ job description.

In a health system, as in any other system, one has to identify the problems before they can be solved. Without some form of audit problems can not be identified effectively. The basic assumption of audit and feedback is that once the problems have been identified and made known to health professionals there will be a self-correction in behaviour and the quality of care will improve. This is the so-called 'Hawthorne effect', which postulates that care improves if the care providers know that their performance is being observed. The overall small effect of audit and feedback found in the review is surprising, and stresses that a health system is more like an ecosystem where the interactions are complex and results often unpredictable, rather than a machine where results can be readily predicted. There are many factors that impact on care, such as staffing levels, staffing morale, availability of facilities and levels of knowledge. These factors must be taken into consideration when attempting to change the behaviour of health-care professionals.

2.2. Applicability of the results

Audit and feedback is usually part of clinical governance (systematic way of monitoring and improving patient care in hospitals), and all health professionals are usually required to perform some sort of audit and feedback. As audit and feedback is being performed anyway, any effort made to make it effective would be worthwhile. Besides, this approach is inexpensive. The review suggests that where the baseline of adherence to recommended practices is low there is a greater likelihood of success with audit and feedback. Since this often the case in under-resourced settings, the findings of this review would be applicable to under-resourced settings. The review also suggests that a key factor in achieving greater impact is the intensity of the feedback. Factors in feedback that have been shown to be effective are: provision of the feedback by a senior person; continued feedback sessions over a long period; face-to-face feedback to individuals; and feedback combined with educational meetings.

Most of the studies were carried out in developed countries with only four being performed in developing countries, (Thailand two, Lao People's Democratic Republic one and Uganda one). However, there is no reason to suppose the factors for feedback found to be effective in the review would not be the same in developing countries. The issue for developing countries is to see how feasible it would be to incorporate those factors in their audit and feedback processes. If there are mortality and morbidity meetings in a health institution, the areas of feedback that have been shown to be effective would be relatively easy to incorporate in the meetings. As stated above, the first step is to institutionalize review meetings, and for this it is essential that health-care professionals regard review of performance as an integral part of their job description.

2.3. Implementation of the intervention

There are three types of audit. A facility audit analyses structures in the health-care system, e.g. staffing or the availability of drugs and equipment. In an outcomes audit the outcome or event is identified and the person performing the audit looks back to see why there was that outcome. The best examples of this are the confidential enquiries into maternal deaths or perinatal deaths review. These are one of the oldest methods of improving perinatal care (2). Importantly, an outcomes audit is easy to perform and does not require external expertise. The third type of audit is a process audit that examines how a particular process is being managed. A process audit (also called clinical audit, criterion-based clinical audit) is used to assess the quality of care in a particular process (3, 4). The studies included this review mainly concerned process audits. There are no randomized trials on the effect of outcomes audit (5). There is descriptive evidence that outcomes audit are associated with reduction in deaths (5). There is no reason to suspect the key areas of feedback found in the studies on process audit would be any different to those on outcomes audit.

Audit and feedback is an essential aspect of the job of any clinic manager. In settings where a review system is not in place a decision will need to be taken regarding the type of audit that will be performed. An outcomes audit is the easiest type of audit to implement. There are excellent guides to performing such audits (6). Maternal and perinatal morbidity and mortality meetings can be effective in improving the quality of care. To strengthen the effect of audit and feedback, these meetings could be conducted by a senior health professional, become a regular feature of professional practice, and be combined with an educational component. Face-to-face feedback to individuals where there is a specific problem could also improve practice. Furthermore, it is possible the effect of these meetings will be strengthened if the health-care professionals are actively involved and have specific and formal responsibility for implementing any change in practice.

Once outcomes audit such as perinatal and maternal morbidity and mortality meetings are established, institutions can progress to process audits to examine how the procedure is being managed. There are guides to implement this as well (3, 4).

3. RESEARCH

There are many questions to still be answered, despite many trials having being conducted. Answering the unanswered questions is particularly important for resource-poor areas as audit and feedback is potentially an inexpensive, simple method of improving the quality of care. This makes research into this area vital for developing countries. Identifying the most effective method of implementing evidence-based care can have a major impact in these countries. Future research should: be well designed, conducted and reported; have the statistical power to detect small changes; be designed to provide insights into the complex dynamics underlying the variable effectiveness of audit and feedback; and make direct comparisons between different ways of doing audit and feedback. It would be also useful to conduct research to understand the relative roles played by culture, organization of health-care within an institution and impact of individual health professional in enhancing or inhibiting the effectiveness of audit and feedback.

References

• Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. The Cochrane Library, Issue 3,;2006.
• Rowe AK, de Savigny D, Lanata CF, Victoria CG. How can we achieve and maintain high-quality performance of health workers in low-resource setting. Lancet 2005;366:1026-1035.
• Wagaarachchi PT, Graham WJ, Penny GC, McCaw-Binns A, Yeboah Antwi K, Hall MH. Holding up a mirror: changing obstetric practice through criterion-based clinical audit in developing countries. Int J Gynaecol Obstet 2001;74:119-130.
• Wagaarachchi PT, Asare K, Ashley D, Gordon G, Graham WJ, Hall MH. Conducting criterion-based clinical audit of obstetric care: a practical field guide. Dugald Baird Centre for Research on Women’s Health,;Aberdeen, 2001.
• Pattinson RC, Bastos MH, Say L, Makin JD. Critical incident audit and feedback to improve perinatal and maternal mortality and morbidity. The Cochrane Library, Issue 3,;2006.
• Lewis G. Confidential enquiries into maternal deaths. In: Beyond the numbers: Reviewing maternal deaths and complications to make pregnancy safer. World Health Organization, Geneva 2004;77-102.

This document should be cited as: Pattinson RC. Audit and feedback: effects on professional practice and health-care outcomes: RHL commentary (last revised: 15 December 2006). The WHO Reproductive Health Library; Geneva: World Health Organization.