The term “Publication Bias” can be used in a variety of different ways and includes problems ranging from non-publication of trials, to delayed publication, publication in less accessible journals or languages, and selective publication of the outcomes in papers that do get published. The problem is that if the results influence where, when and what is published, then it is biased sample of the whole data, and any systematic review of the published results may be misleading.
Human nature is such that Trialists and Sponsors have an agenda in what they publish and the way the data is presented. For example in small trials there are often baseline imbalance between the active treatment and control groups. This can be exploited in the way that the data is presented as change from baseline or absolute post treatment values (for lung function data as an example). Sometimes only the outcome favouring the product is presented, or it may be emphasised over the alternative. An example of this was found in the Systematic review comparing metered dose inhalers in the BMJ (Brocklebank D, Wright J, Cates C. Systematic review of clinical effectiveness of pressurised metered dose inhalers versus other hand held inhaler devices for delivering corticosteroids in asthma. BMJ 2001;323:896-.) An apparent group difference between delivery devices disappeared when the FEV1 data from some of the research papers was adjusted from absolute to change from baseline values (or vice versa).
With care systematic reviewers can be on the lookout for such problems, and sometimes adjust for them. However when outcomes are not published at all, or even whole trials do not reach the public domain, this represents a serious threat to the combined results of clinical trials that are collected together in Systematic Reviews.
A particularly worrying example of this was published this year in the Lancet.
Here is the abstract and reference:
Abstract
“BACKGROUND: Questions concerning the safety of selective serotonin reuptake inhibitors (SSRIs) in the treatment of depression in children led us to compare and contrast published and unpublished data on the risks and benefits of these drugs. METHODS: We did a meta-analysis of data from randomised controlled trials that evaluated an SSRI versus placebo in participants aged 5-18 years and that were published in a peer-reviewed journal or were unpublished and included in a review by the Committee on Safety of Medicines. The following outcomes were included: remission, response to treatment, depressive symptom scores, serious adverse events, suicide-related behaviours, and discontinuation of treatment because of adverse events. FINDINGS: Data for two published trials suggest that fluoxetine has a favourable risk-benefit profile, and unpublished data lend support to this finding. Published results from one trial of paroxetine and two trials of sertraline suggest equivocal or weak positive risk-benefit profiles. However, in both cases, addition of unpublished data indicates that risks outweigh benefits. Data from unpublished trials of citalopram and venlafaxine show unfavourable risk-benefit profiles. INTERPRETATION: Published data suggest a favourable risk-benefit profile for some SSRIs; however, addition of unpublished data indicates that risks could outweigh benefits of these drugs (except fluoxetine) to treat depression in children and young people. Clinical guideline development and clinical decisions about treatment are largely dependent on an evidence base published in peer-reviewed journals. Non-publication of trials, for whatever reason, or the omission of important data from published trials, can lead to erroneous recommendations for treatment. Greater openness and transparency with respect to all intervention studies is needed.”
Reference: Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data.
Lancet 2004;363(9418):1341-5.
The Editor of the Lancet has expressed strong views about the way the data was withheld.
(Depressing research. Lancet 2004;363(9418):1335.)
The public prosecutor in New York is taking one of the manufacturers of an SSRI to court over this issue. If the case is successful perhaps this will provide a lever for change and a greater level of transparency in papers reporting the results of medical research.
(Is GSK guilty of fraud? Lancet 2004;363(9425):1919.)
The Moral of the Story
I think we have to assume that publication bias is always present (rather than the opposite) and the tests that we carry out, such as funnel plots, should be looked at to see if there is reasonable evidence that publication bias is NOT present!
The problem is particularly acute for the reporting of adverse events, as in the example above. There are already moves afoot to encourage the registration of all clinical trials, but this will not address the problem of selective reporting of results from trials (for example adverse event recording). I wonder if it would help to ask that when the results of controlled trials are analysed, the groups should be concealed from the statistician and authors so that significant differences are reported as such, and the direction of effect (for or against the active treatment) is then added later. Perhaps consideration should be given to including such an approach into the next revision of the Consort statement?