Evidence based medicine – historical perspective and critique

Mr James W Fairley BSc MBBS FRCS MS
Consultant ENT Surgeon

Last updated 23 July 2007
© 1990 – 2016 JW Fairley

Philosophical, scientific & statistical background to evidence based medicine

Historical perspective and critique

Limitations of EBM in surgical specialities

The use and abuse of Evidence Based Medicine: Towards Knowledge Based Medicine

The concept of focal sepsis in the sinuses: An historical caveat

Thesis submitted for the degree of Master of Surgery, University of London, 1993.


Correlation of nasal symptoms with objective findings and surgical outcome measurement.

Historical basis of Evidence Based Medicine

Medical practice based on evidence is nothing new, as the authors of EBM textbooks readily point out (Sackett et al, 2000). The difference is that modern “Evidence Based Medicine” is explicit about the kind of evidence we are using, and the evidence is rated into a hierarchy of different levels.

Table 1.     Hierarchy of EBM (Evidence Based Medicine)
simplified from CEBM Oxford 2001 (see full detailed table – opens new window)
Level    Type of Evidence
I Systematic Reviews of well controlled Randomized Controlled Trials (meta-analysis) or single RCT with narrow CI (confidence interval)
II Systematic review cohort studies or lesser quality RCTs
III Case controlled studies (non randomized)
IV Case series (no control group)
(V) Expert opinion (GOBSAT – Good Old Boys Sat Around Table)

The lowest rating is given to the evidence most frequently used in practice. Advice from experts working in the field is relegated to the derogatory GOBSAT. The highest rating is accorded to meta-analysis of well controlled randomized clinical trials. Historical precedent – “we do it that way because that’s the way we’ve always done it” – is not even accorded a rating.

The justification for this hierarchical evidence base is that

  1. Half of what we know will be wrong within ten years, and
  2. We don’t know which half.

Historical nihilism vs. accumulated professional wisdom

I don’t think the second part of the justifation is correct. In fact, I believe that experienced doctors have a fairly good idea of what is likely to remain valid long term. For example, if you want to cure someone with any serious disease you should pay attention to their nutrition. Unless humans evolve to an extracorporeal existence, it is unlikely this principle will change in future.

The refusal to give due weight to accumulated professional wisdom is a peculiarly modern Western scientific variety of historical nihilism, and an unattractive quality of EBM. The advice given by Sackett et al (2000) to
“burn your traditional textbooks”
belittles the efforts and achievements of our predecessors. If the past really is another country, the proponents of EBM stand accused of cultural elitism if not outright racism toward its inhabitants.

Of course it is right to question, and where possible test historical precedent, but to give it no place at all in the hierarchy of evidence is throwing out the baby with the bath water.

  • Traditional ways of doing things in an ancient profession like surgery don’t get there by accident.
  • Tradition incorporates the distilled wisdom from countless, many long forgotten, practical trials.
  • Errors made by previous generations are, in general, weeded out by the slow process of evolution.

Some aspects may be proven wrong, but there are some fundamental aspects of looking after patients that are unlikely to change.

Furthermore, advances that do occur tend to come from the application of new technology, not by sewing doubt over basic principles.

Has the fundamental assumption underlying hierarchical evidence based medicine been tested scientifically?

The hierarchy of evidence relies on an assumption. The assumption is that professionals don’t know which parts of their practice are soundly based, and which are not. Are professionals really incapable of judging the wheat from the chaff in the knowldedge base?

This assumption of professional ignorance forms the fundamental basis for applying RCT’s in all areas of medicine. It is probably misguided. It could, in fact, be tested by scientific experiment, following Popperian logic.

If a group of experienced doctors were to draw up a selected list of what they believe will still be correct in ten years time, I would predict that they would do considerably better than 50%. The null hypothesis (as espoused by the advocates of hierarchical EBM) would predict only a 50% success rate.

Although there is plenty of anecdotal evidence of the benefits of EBM, I am not aware of any systematic scientific experimental basis which supports the assumption of professional ignorance that underlies the hierarchy of EBM. In particular, I am not aware of any such study in any surgical speciality.

EBM in Victorian times

Our Victorian forebears practised evidence based medicine. Their evidence came from detailed clinical description, correlating the symptoms and signs in life with the pathological findings – often post-mortem, since the diseases they dealt with were serious and the therapeutic options limited. Like today, the accumulation of personal experience and skill lay at the core of clinical practice, together with a lively exchange of views at national and international meetings, and a regular series of published case reports, reviews and polemic in the medical literature of the day. Systematic reporting of clinical outcomes was, however, rare. Most published series would comprise a collection of case reports. These served the dual purpose of a reference database – “How Mr X dealt with a similar case” – and advertising the experience of the author to his colleagues. La plus ça change …

EBM 2007: What does a meta-analyst do?

Among serious academic doctors today, the case report – and its big brother, the case series – is out of favour. Modern evidence based medicine is meant to be based on soundly designed trials, with enough patients to test the hypothesis to a known degree of statistical power. Meta-analysis then pools the results from several trials.

It also allows the meta-analyst to do research entirely on the back of other people’s work – in fact without ever leaving the computer screen, and without the messy business of actually seeing patients.

To give such efforts their due, there has been, over the last fifteen years, a useful formalization of rules for what constitutes best evidence, and a genuine effort to identify, disseminate and apply it. This is relatively easy for drug trials, but much more difficult for surgical treatments where the skill and experience of the individual surgeon is such a major factor influencing outcome.

Furthermore, in conditions such as rhinosinusitis, where

  • the condition being treated is not life-threatening
  • does not have simple objective parameters to measure, and
  • is subject to spontaneous remission and relapse

the assessment of outcome is itself very difficult.

I know this from personal experience of designing and carrying out a randomised controlled trial of endoscopic sinus surgery versus conventional inferior meatal antrostomy. At the time of designing the trial (in the late 1980’s) there were no generally accepted outcome measures and I was obliged to develop and validate my own. I also carried out all the operations personally, thus avoiding some of the difficulties in controlling for surgical skill. The 1993 results of this trial were accepted as meeting the criteria for inclusion in the evidence base by the unbiased systematic review process of the Cochrane collaboration in their review of FESS, 2006.

In all areas of medicine, there is a vast published literature. Unfortunately, even today, very little of this fulfils the criteria for allowing good evidence based decisions on diagnosis, prognosis or treatment. The sheer volume of material makes it impossible for all specialists to be up-to-date in all areas. Limited health resources are wasted by continuing ineffective practices, while delays in adopting effective treatments deny potential benefit to other patients. When it comes to appraising the published evidence, no individual has the time, and few have the skills, to do their own systematic reviews. This is where organizations such as the Cochrane collaboration come in.


Who was Cochrane?

Archie Cochrane (1909 – 1988) a British epidemiologist, wrote his seminal work “Effectiveness and Efficiency: Random Reflections on Health Services” in 1972. Cochrane challenged the dominant medical authorities of the day to prove that what they did made any real difference. His field research in South Wales in the 1950’s and 60’s identified the relatively small impact of clinical medicine in those days on health outcomes, and he criticised the lack of scientific methods in clinical practice. The following quotation from Cochrane himself shows that he certainly could not be accused of being a purely academic doctor, remote from the messy business of actually seeing patients:

“The change in the tuberculosis world between 1944 when I was burying my POW tuberculous patients in Germany and the present day when TB deaths are the subject of a special investigation, as in theory they should not happen, is one of the most cheering things I have experienced in my life. The way in which the new treatments and preventive measures were introduced can also serve as a model for the introduction of all new treatments in the future. RCT’s were used from the very beginning….”

The Cochrane collaboration

Following much discussion in medical journals in the 1980’s over matters such as publication bias (the tendency for trials with positive results to be published while those with negative results would be ignored) the Cochrane Collaboration was established in 1992. It uses systematic review and meta-analysis to synthesise the best available evidence in an unbiased way. Progress has been made in both the scientific methodology of conducting reviews, and in their dissemination for use by health professionals. The output, comprising Cochrane systematic reviews, abstracts of other published systematic reviews and a database of randomised controlled trials, is published on the internet. The work of the Collaboration is carried out by international multi-disciplinary Review Groups. Members include clinicians, scientists, patients and their carers, policy makers and others, with an editorial group. Specialist Cochrane Review Groups around the world cover most areas of health care.

Cochrane ENT group

A Cochrane Ear, Nose and Throat Disorders Group was established in September 1998 in Oxford. It aims to carry out systematic reviews of evidence on the prevention, treatment and rehabilitation of ear, nose and throat disorders, including head and neck cancer. Unfortunately, most in most areas of clinical practice, the Cochrane reviews are hampered by a lack of quality RCT’s as their raw material. Therefore most reviews to date have concluded “no evidence of benefit” and “more studies required”.

My Review of the usefulness of Cochrane Reviews

In May 2007, I downloaded the summaries of all 42 published reviews on the Cochrane ENT group website for the eight year period 1999 to 2006, and classified them into:


  • Positive conclusions – where there was sufficient evidence to make a treatment recommendation for the intervention in question

  • Negative conclusions, where there was insufficient evidence.

I then divided the interventions which formed the subjects of the reviews into


  • Surgical – reviews of surgical operations

  • Medical – reviews of non-surgical interventions, including drug trials, physical therapies, screening.

The results are shown graphically in Figures 1 and 2, the data in Table 2.

Figure 1
Conclusions from eight years of publication of Cochrane Reviews of ENT surgical interventions 1999 – 2006

Chart Cochrane Review Conclusions
Eight years of publication 1999 - 2006
ENT Surgical interventions
(n = 10)

The lack of RCT’s is particularly noticeable for ENT surgical interventions. Only ten reviews of surgical interventions have been published in eight years, of which only one reached a positive conclusion. The only intervention for which good RCT evidence was found in the literature was grommets (ventilation tubes) insertion. The conclusion, rather grudging in tone, follows:

"Evidence suggests grommets only offer a short-term hearing improvement in children with simple glue ear (otitis media with effusion or OME) and no other serious medical problems. No effect on speech and language development has been proven."

Figure 2
Conclusions from eight years of publication of Cochrane Reviews of ENT medical interventions 1999 – 2006

Chart Cochrane Review Conclusions
Eight years of publication 1999 - 2006
ENT medical interventions
(n = 32)

More than three times as many Cochrane reviews have been published on ENT medical interventions compared to surgical.
A higher proportion of reviews of medical interventions found sufficient RCT evidence to come up with a positive conclusion.
Almost half the reviews of medical interventions found sufficient evidence, compared with only one in ten for surgical interventions.


Table 2
Cochrane ENT Group systematic reviews published 1999 – 2006 (n = 42)
Classified by intervention type (medical or surgical) and conclusion (positive or negative)

















































































































































































































































































































Year

Intervention type

Subject

Conclusion – short

Conclusion – long

1999

S

Perioperative local anaesthesia for reducing pain following tonsillectomy

N

There is no evidence that the use of perioperative local anaesthetic in patients undergoing tonsillectomy improves post-operative pain control. The trials identified were of small size and several involved the perioperative co-administration of intravenous opiates which may have masked any beneficial effect of the local anaesthetic. Further randomised controlled trials are necessary.

1999

S

Tonsillectomy versus non-surgical treatment for chronic/recurrent acute tonsillitis

N

The effectiveness of tonsillectomy has not been formally evaluated. Further trials addressing relevant outcome measures are required.

2000

M

Betahistine for Ménière’s disease or syndrome

N

Not enough evidence about the effects of Betahistine on Ménière’s disease or syndrome

2001

S

Dissection versus diathermy for tonsillectomy

N

Not enough evidence to show the best way to remove tonsils surgically.

2001

S

Surgical versus non-surgical interventions for vocal cord nodules

N

Not enough evidence to compare surgical with non-surgical techniques to help remove vocal cord nodules.

2002

S

Radiotherapy versus open surgery versus endolaryngeal surgery (with or without laser) for early laryngeal squamous cell cancer

N

Not enough evidence on which treatment is best for early stages of cancer in the vocal cords that has not spread to the voice box.

2002

M

Steroids for improving recovery following tonsillectomy in children

P

A single dose of corticosteroids while on the operating table could prevent post-operative vomiting for many children having their tonsils removed, without adverse effects.

2003

M

Interventions for recurrent idiopathic epistaxis (nosebleeds) in children

N

More research is needed to show the best options for reducing nosebleeds of unknown cause in children.

2003

M

Ear drops for the removal of ear wax

P

Using ear drops to remove impacted ear wax is better than no treatment, but no particular sort of drops can be recommended over any other.

2003

M

Sublingual immunotherapy for allergic rhinitis

P

Sublingual immunotherapy can relieve allergic rhinitis (including hay fever), although it is not known whether it is as effective as injections or nasal immune treatments.

2004

M

Antibiotic prophylaxis in clean and contaminated ear surgery

N

There is no strong evidence that the large scale use of prophylactic antibiotics in ear surgery is helpful in reducing postoperative complications.

2004

M

Ginkgo biloba for tinnitus

N

No evidence that Ginkgo biloba is effective for tinnitus. Further research is needed.

2004

S

Interventions for acute auricular haematoma

N

There is no good evidence to suggest which method of treatment is best in the treatment of acute auricular haematoma.

2004

S

Surgical interventions for pharyngeal pouch

N

There is no evidence from high quality randomised controlled trials to demonstrate the effectiveness of endoscopic compared with open procedures for pharyngeal pouch. There is no good evidence to establish whether one endoscopic procedure is superior to another.

2004

M

Botulinum toxin injections for the treatment of spasmodic dysphonia

P

Botulinum toxin has been shown to benefit some aspects of voice production in speakers with spasmodic dysphonia.

2004

M

Scopolamine for preventing and treating motion sickness

P

Scopolamine is effective for preventing motion sickness.

2004

M

The Epley (canalith repositioning) manoeuvre for benign paroxysmal positional vertigo

P

The Epley manoeuvre can help spinning and dizziness on moving the head (benign paroxysmal positional vertigo) in the short term but more research is needed.

2004

S

Grommets (ventilation tubes) for hearing loss associated with otitis media with effusion in children

P

Evidence suggests grommets only offer a short-term hearing improvement in children with simple glue ear (otitis media with effusion or OME) and no other serious medical problems. No effect on speech and language development has been proven.

2005

M

Acid reflux treatment for hoarseness

N

There is not enough evidence that anti-reflux therapies are effective in treating hoarsenes

2005

M

Adjuvant antiviral therapy for recurrent respiratory papillomatosis

N

The review found no good quality trials demonstrating the effectiveness or otherwise of antivirals in recurrent respiratory papillomatosis

2005

M

Hyperbaric oxygen as an adjuvant treatment for malignant otitis externa

N

There is no clear evidence to demonstrate the effectiveness of hyperbaric oxygen therapy in the treatment of malignant otitis externa. Further research is required.

2005

M

Steroids for idiopathic sudden sensorineural hearing loss

N

The value of steroids in the treatment of idiopathic sudden sensorineural hearing loss remains unclear since the evidence obtained from randomised controlled trials are contradictory in outcome, in part because the studies are based upon too small a number of patients.

2005

M

Universal neonatal hearing screening versus selective screening as part of the management of childhood deafness

N

Insufficient evidence to establish the long-term effectiveness of universal screening versus selective screening programmes for childhood deafness

2005

M

Systemic antibiotics versus topical treatments for chronically discharging ears with underlying eardrum perforations

P

Topical quinolone antibiotics can clear aural discharge better than systemic antibiotics; topical non-quinolone antibiotic (without steroids) or antiseptic results are less clear. Evidence regarding safety was weak. Further studies should clarify topical non-quinolones and antiseptic effectiveness, assess longer-term outcomes (for resolution, healing, hearing, or complications), and include further safety assessments, particularly to clarify the risks of ototoxicity and whether there may be fewer adverse events with topical quinolones than other topical or systemic treatments.

2005

M

Topical antibiotics without steroids for chronically discharging ears with underlying eardrum perforations

P

Topical quinolone antibiotics can clear aural discharge better than no drug treatment or topical antiseptics; non-quinolone antibiotic effects (without steroids) versus no drug or antiseptics are less clear. Studies were also inconclusive regarding any differences between quinolone and non-quinolone antibiotics, although indirect comparisons suggest a benefit of topical quinolones cannot be ruled out. Further trials should clarify non-quinolone antibiotic effects, assess longer-term outcomes (for resolution, healing, hearing, or complications) and include further safety assessments, particularly to clarify the risks of ototoxicity and whether quinolones may result in fewer adverse events than other topical treatments.

2006

M

Antidepressants for patients with tinnitus

N

There is insufficient evidence to say that antidepressant drug therapy improves tinnitus.

2006

M

Antihistamines and/or decongestants for otitis media with effusion (OME) in children

N

Because the pooled data demonstrate no benefit and some harm from the use of antihistamines or decongestants alone or in combination in the management of OME, we recommend against their use.

2006

M

Autoinflation for hearing loss associated with otitis media with effusion

N

All of the studies were small, of limited treatment duration and short follow up. However, because of the low cost and absence of adverse effects it is reasonable to consider autoinflation whilst awaiting natural resolution of otitis media with effusion. Further research should consider the duration of treatment and the long-term impact of autoinflation on developmental outcomes in children.

2006

M

Capsaicin for allergic rhinitis in adults

N

Not enough evidence on the effectiveness of capsaicin in treating allergic rhinitis.

2006

M

Diuretics for Ménière’s disease or syndrome

N

There is insufficient good evidence of the effect of diuretics on vertigo, hearing loss, tinnitus or aural fullness in clearly defined Ménière’s disease.

2006

M

Hyperbaric oxygen for idiopathic sudden sensorineural hearing loss and tinnitus

N

Hyperbaric oxygen may improve deafness and tinnitus after sudden hearing loss of unknown cause, but the evidence is of poor quality.

2006

M

Interventions for ear discharge associated with grommets (ventilation tubes)

N

More research is needed to find the most effective treatment for discharge from ears in patients who have had grommets fitted.

2006

M

Interventions to promote the wearing of hearing protection

N

Limited evidence does not show whether tailored interventions are more or less effective than general interventions in workers, 80% of whom already use hearing protection.

2006

M

Topical nasal steroids for intermittent and persistent allergic rhinitis in children

N

The three included trials provided some weak and unreliable evidence for the effectiveness of Beconase® and flunisolide used topically intranasally for the treatment of intermittent and persistent allergic rhinitis in children. The reduction of severity in symptoms as assessed by the trialists could not be confirmed with the data provided and decisions on the use of these medications should, until such time as more robust evidence is available, be guided by the physician’s clinical experience and patients’ individual circumstances and preferences.

2006

S

Functional endoscopic sinus surgery for chronic rhinosinusitis

N

The limited evidence available suggests that FESS as practiced in the included trials does not confer additional benefit to that obtained by medical treatment (+/- sinus irrigation) in chronic rhinosinusitis. More randomised controlled trials comparing FESS with medical and other treatments, with long-term follow up, are required.

2006

S

Identification of children in the first four years of life for early treatment for otitis media with effusion

N

There is no evidence of important benefits of early identification of otitis media with effusion in children in their first four years of life in developed countries.

2006

M

Immunotherapy by allergen injections for seasonal allergic rhinitis (‘hay fever’)

P

This review has shown that specific allergen injection immunotherapy in suitably selected patients with seasonal allergic rhinitis results in a significant reduction in symptom scores and medication use. Injection immunotherapy has a known and relatively low risk of severe adverse events. We found no long-term consequences from adverse events.

2006

M

Chemotherapy as an adjunct to radiotherapy in locally advanced nasopharyngeal carcinoma

P

Chemotherapy led to a small but significant benefit for overall survival and event-free survival. This benefit was essentially observed when chemotherapy was administered concomitantly with radiotherapy.

2006

M

Cognitive behavioural therapy for tinnitus

P

We did not find a significant difference in the subjective loudness of tinnitus, or in the associated depression. However we found a significant improvement in the quality of life (decrease of global tinnitus severity) of the participants, thus suggesting that cognitive behavioural therapy has an effect on the qualitative aspects of tinnitus and contributes positively to the management of tinnitus.

2006

M

House dust mite avoidance measures for perennial allergic rhinitis

P

There is limited evidence that reducing house dust mites might improve symptoms of allergic rhinitis, but more research is needed to clarify the effectiveness of acaricides both as a mono-intervention and as part of a more multi-faceted intervention incorporating high efficiency particulate air (HEPA) filters and allergy control bedding.

2006

M

Oral or topical nasal steroids for hearing loss associated with otitis media with effusion in children

P

Both oral and topical intranasal steroids alone or in combination with an antibiotic lead to a quicker resolution of OME in the short term, however, there is no evidence of longer term benefit.

2006

M

Oral steroids for nasal polyps

P

The authors found one small randomised controlled trial, albeit of poor quality, that suggests a short-term effect of oral steroids in patients with multiple nasal polyps. To address the issue more thoroughly a well designed, prospective randomised controlled trial is needed.

My Conclusion

To date, the Cochrane Systematic Review methodology, with its strict reliance on RCT’s, has not been very useful in assessing the effectiveness of ENT surgical interventions.

Other ways of assessing the effectiveness of surgical interventions

Some of the commonest ENT operations (tonsillectomy for example) apparently have no valid evidence base according to EBM criteria, yet their success is attested by very high patient satisfaction rates (as shown on the Scottish National Tonsillectomy Audit). Either

  • the randomized controlled studies have to be done, or
  • we have to allow that there is more than one way to evaluate the effectiveness of some treatments.

Before any comparative studies of surgical interventions can be done, there are great difficulties to be overcome in terms of

  • defining some of the conditions we treat
  • defining valid and reliable outcome measures
  • allowing for the effects of individual surgical skill in those studies involving surgical treatment.

Further EBM pages authored by JW Fairley

Philosophical, scientific and statistical basis of Evidence based medicine

Limitations of evidence based medicine – should Cochrane reviews of surgical interventions concluding “no evidence of benefit” come with a health warning?

Previous EBM-related publications by JW Fairley

Fairley JW (1990) Patrick Watson-Williams and the concept of focal sepsis in the sinuses: An historical caveat for functional endoscopic sinus surgery

Fairley JW (1993) Correlation of nasal symptoms with objective findings and surgical outcome measurement.

Bibliography

Popper KR 1959 The logic of scientific discovery. Hutchinson & Co, London. 8th Impession 1975 ISBN 0 09 111721 6

Cochrane AL 1972 Effectiveness And Efficiency: Random Reflections on Health Services. Facsimile Edn, additional contributions Silagy C, Chalmers I, 1999 RSM Press ISBN 185315394X

Sackett D.L., Straus S.E., Richardson W.S, Rosenberg W., Haynes R.B. Evidence Based Medicine: How to practice and teach EBM 2nd Edn 2000 Churchill Livingstone, London ISBN: 0 443 06240 4

Disclaimer

All information and advice on this website is of a general nature and may not apply to you. There is no substitute for an individual consultation. We recommend that you see your General Practitioner if you would like to be referred.