publication-bias-funnel-plots

Publication Bias and Funnel Plots: A Plain-Language Guide for Evidence Reviewers

If every study ever conducted on a question were published, reviewing the evidence would be much easier. In reality, studies with exciting, statistically significant results are more likely to be written up, accepted by journals, published quickly and cited often. Studies that find nothing tend to sit in drawers. The result is that the visible literature can be systematically more optimistic than the truth. This is publication bias, and it is one of the most important threats to any literature review or meta-analysis. This guide explains how it arises, how funnel plots and related methods try to detect it, what those methods can and cannot tell you, and how a source-linked research workbench such as Yila can help you document what you looked for and what you found.

How publication bias happens

Publication bias is not one thing but a family of related processes, sometimes grouped under the term reporting biases.

• Whole-study non-publication. A trial is completed, finds no effect, and is never submitted.
• Time-lag bias. Positive studies are published faster, so early reviews over-represent them.
• Selective outcome reporting. A study measures ten outcomes and reports the three that were significant.
• Language and location bias. Positive results may be more likely to appear in English-language, high-visibility journals, while null results appear in local journals or theses.
• Citation bias. Positive studies are cited more, so they are easier to find through citation chasing.

Each mechanism pushes the visible evidence in the same direction. A review that only searches major databases for published articles will usually inherit that push.

Prevention beats detection

The most effective response to publication bias is to look harder for unpublished evidence. Practical steps include:

• searching trial registries such as ClinicalTrials.gov and the WHO ICTRP for completed studies without publications;
• searching for theses, dissertations, conference abstracts and preprints;
• checking regulatory documents and clinical study reports where relevant;
• contacting authors and experts to ask about unpublished or ongoing work;
• comparing registered protocols with published papers to identify missing outcomes.

Every one of these steps should be recorded in your search log. An evidence-traceable workbench makes this easier because each source you check, including the ones that return nothing, can be logged alongside the question it relates to. Absence of evidence is only meaningful if you can show where you looked.

What a funnel plot shows

A funnel plot is a scatter plot of study results against a measure of their precision, usually the standard error. Each dot is a study. The horizontal axis shows the effect estimate; the vertical axis shows precision, with the most precise (usually largest) studies at the top.

In the absence of bias, the plot should look like an inverted funnel. Large, precise studies cluster tightly near the true effect at the top. Smaller, less precise studies scatter more widely at the bottom, but symmetrically on both sides of the true effect, because chance pushes them in either direction equally.

When small studies with unfavourable results are missing, one lower corner of the funnel is empty. The plot looks lopsided, with small studies bunched on the side of a larger effect.

Why asymmetry is not proof of publication bias

This is the most important point in the whole topic. Funnel plot asymmetry has several possible causes, and publication bias is only one of them.

Genuine heterogeneity. Small studies may differ systematically from large ones. They may enrol higher-risk patients, deliver the intervention more intensively, or be run by the original developers. In those cases, small studies might genuinely show larger effects.

Methodological quality. Smaller studies are often less rigorous, and less rigorous studies tend to report larger effects. Asymmetry may reflect bias within studies rather than missing studies.

Artefacts. Some effect measures, such as odds ratios with rare events, are mathematically correlated with their standard errors, which can create asymmetry on their own.

Chance. With fewer than about ten studies, funnel plots are very hard to interpret. Random variation can produce apparent asymmetry or hide real asymmetry.

For these reasons, methodologists generally talk about small-study effects rather than publication bias when describing asymmetry. The question is then why small studies differ, and publication bias is one candidate explanation.

Statistical tests and adjustments

Several tests formalise the visual inspection. Egger's regression test is the best known; it tests whether the relationship between effect size and precision departs from what symmetry would predict. Variants exist for binary outcomes and for different effect measures. All these tests have low power when the number of studies is small, so a non-significant result does not rule out bias.

Adjustment methods try to estimate what the pooled effect would be without the bias. Trim-and-fill imputes hypothetical missing studies to make the funnel symmetric. Selection models estimate the probability that studies with different p-values were published. Regression-based approaches such as PET-PEESE extrapolate to a hypothetical study with perfect precision. These methods can be useful as sensitivity analyses, but they rest on strong assumptions and frequently disagree with each other. Treat their results as an indication of how fragile the main estimate might be, not as a corrected answer.

Reporting what you found

A transparent review describes publication bias in three parts.

1. What was done to minimise it. Registries searched, grey literature sources, author contacts.
2. What was done to detect it. Funnel plots, tests and their results, with the number of studies included.
3. How it affects conclusions. Whether certainty in the evidence was downgraded, and whether sensitivity analyses changed the picture.

Under the GRADE approach, suspected publication bias is one of the domains that can lower certainty in a body of evidence. Even when formal tests are not possible, reviewers can judge the risk from indirect signs: a field dominated by small industry-funded trials, registered studies with no results, or a sudden absence of null findings.

Beyond meta-analysis

Publication bias matters for narrative reviews, scoping reviews and everyday desk research too, even when no funnel plot can be drawn. If your evidence comes mostly from press releases, conference highlights or vendor case studies, you are looking at a heavily filtered sample. Say so. Look for independent evaluations and negative findings deliberately, and record the search.

AI research tools can make this better or worse. A tool that summarises the most-cited papers will amplify citation bias. A tool that shows exactly which sources it used, and lets you add registries and grey literature to the evidence pool, can help you see the gaps. The research agent at yila.ai keeps every summarised claim linked to its origin, which makes it easier to notice when a conclusion rests entirely on one type of source.

Checklist

• Trial registries and grey literature searched, with dates and results logged.
• Registered protocols compared with published outcomes where possible.
• Funnel plot produced only when there are enough studies, and interpreted cautiously.
• Asymmetry discussed as small-study effects, with alternative explanations considered.
• Adjustment methods used as sensitivity analyses, not as corrections.
• Certainty of evidence adjusted where bias is suspected.

Final thoughts

You cannot review evidence that you never see. Publication bias is the reason that searching beyond the obvious, recording where you looked, and interpreting funnel plots with humility all matter. Treat asymmetry as a question rather than an answer, and make your search for missing evidence as visible as the evidence you found.