Interpreting evidence and analysing data

• Select descriptive summaries appropriate to the distribution and question.
• Interpret uncertainty, confidence intervals and statistical significance without common overclaims.
• Recognise confounding, selection bias, information bias and regression to the mean.
• Use qualitative evidence systematically and appreciate triangulation.
• Distinguish statistical evidence from clinical, operational and patient importance.

Analysis should answer the pre-specified question. Begin by checking the denominator, missingness, units, impossible values and whether the data represent the intended population. Sophisticated statistical output cannot rescue weak definitions or biased sampling.

Create a simple analysis plan before looking for desirable results. State the primary comparison or descriptive objective, key subgroups, handling of missing data and planned visual displays. Exploratory analysis is legitimate, but label it as exploratory.

For improvement work, analysis often focuses on patterns over time and practical signal. For formal research, inferential analysis may require a more rigorous model, assumptions and pre-specified methods.

Always understand the denominator. A fall in incidents can reflect fewer opportunities, under-reporting or real improvement. Rates often allow fairer comparison across changing activity volumes, but only if the exposure measure is meaningful.

Percentages from very small denominators can appear dramatic. Report counts alongside percentages. Where repeated measures involve the same people or units, recognise that observations may not be independent.

In equity analyses, disaggregation can reveal differences hidden by an overall average, but small cells may threaten confidentiality and create unstable estimates.

The mean uses every value but is sensitive to extreme observations. The median describes the middle observation and is often more informative for skewed data such as length of stay or waiting time. Report the distribution rather than selecting a summary statistic by habit.

Variation is not noise to be ignored. Range, interquartile range and standard deviation describe different aspects of spread. In improvement, run charts and statistical process control place variation in time order, preserving information that a before/after average can destroy.

Ask whether a change in average matters operationally. A two-minute reduction in a pathway may be statistically detectable in a huge dataset but clinically irrelevant.

A confidence interval expresses uncertainty around an estimate under the assumptions of the method. Wider intervals indicate less precision. Interpretation should focus on the range of plausible effects, not merely whether the interval crosses a conventional null value.

For an intervention, an interval may include both clinically worthwhile benefit and trivial effect; that is a different conclusion from an interval that tightly excludes meaningful benefit.

Do not describe a 95% confidence interval as a 95% probability that the fixed true value lies inside this particular interval. The conventional frequentist interpretation concerns the long-run performance of the interval procedure.

A p-value is not the probability that the null hypothesis is true, nor the probability that the finding occurred “by chance”. It describes how compatible the observed data are with a specified model under the null hypothesis.

The conventional p<0.05 threshold is not a boundary between truth and falsehood. Results should be interpreted with effect size, uncertainty, design quality, multiplicity, prior evidence and practical importance.

Avoid “non-significant therefore no effect”. A study may simply be too imprecise to distinguish important benefit from harm.

Selection bias arises when inclusion or follow-up creates systematic differences relevant to the outcome. Information bias arises from inaccurate or differential measurement. Confounding occurs when a third factor is related to both exposure and outcome and distorts their relationship.

Adjustment methods can reduce measured confounding but cannot guarantee removal of unmeasured confounding. Randomisation helps balance confounders on average but does not fix poor measurement or attrition.

In QI and evaluation, concurrent policy or staffing changes may act as confounders. Annotating time-series charts and maintaining a project log helps interpretation.

Projects are often launched when performance is unusually poor. Even without intervention, an extreme value may move closer to its usual level on the next measurement. This regression-to-the-mean effect can be mistaken for success.

Using multiple baseline points and displaying data over time reduces this risk. Where feasible, use comparison groups or interrupted time-series methods for stronger evaluation.

Do not choose the worst month as “baseline” simply because it makes improvement look larger.

Qualitative analysis involves systematic interpretation of textual, audio or observational material. For thematic analysis, familiarise yourself with the data, code meaningfully, develop and refine themes, check them against the dataset and report the analytic process.

A theme is not merely a frequently used word. It should capture a meaningful pattern relevant to the question. Negative or deviant cases can strengthen interpretation by showing where the dominant explanation does not fit.

Maintain an audit trail of coding and analytic decisions. Where more than one analyst is involved, discussion can improve reflexivity; mechanical “inter-rater agreement” is not always the appropriate marker of qualitative rigour.

Different data sources can answer different parts of a problem. A run chart may show improved waiting time, interviews may reveal increased clerical burden, and complaints may identify a subgroup experiencing worse access. These findings are not contradictory noise; together they describe the system more completely.

Triangulation can examine convergence, complementarity and divergence across sources. Divergence should prompt investigation rather than selective reporting.

Decision-makers need the whole pattern: magnitude, uncertainty, mechanism, experience, equity and feasibility.

Rewrite this statement: “The intervention was successful because waiting time fell from 42 to 34 minutes and p=0.03.” A stronger interpretation would ask about the distribution, time trend, sample size, clinical importance, balancing measures, concurrent changes and uncertainty.

Use the Stage 4 Evidence Interpretation worksheet to record the claim, supporting data, plausible alternative explanations, limitations and decision implications. This forces analysis to serve reasoning rather than decoration.

• Reporting percentages without denominators.
• Equating p<0.05 with clinical importance or proof.
• Calling p>0.05 “no difference”.
• Using a single extreme baseline point.
• Ignoring missing data or subgroup inequity.

1. Institute for Healthcare Improvement (IHI): Model for Improvement; Testing Changes; Establishing Measures; Implementing Changes. https://www.ihi.org/library/model-for-improvement
2. EQUATOR Network: SQUIRE 2.0 reporting guideline for healthcare quality improvement. https://www.equator-network.org/reporting-guidelines/squire/

Educational reference. Verify current local governance requirements before starting a project.