Right now, in the UK and across the world, vaccines for COVID-19 are being rolled out. You may have already received, or be expecting a vaccination offer sometime soon. Vaccines are thought to be our main hope to control the COVID-19 pandemic. Their use has only been possible following robust and rigorous clinical trials, which have demonstrated that they meet high safety and effectiveness standards set by the UK medicines regulator (the MHRA).

Results from COVID-19 vaccine trials have been widely reported in the media. As a clinical trial statistician, I was interested to find out what members of the public thought about the results that were reported, and whether the numbers presented in the media on vaccine effectiveness had been fully understood.

On 29th January 2021, the HEALTHY STATS public involvement group met online to discuss the results reported from two COVID-19 vaccine trials:

- The Pfizer/Biontech COVID-19 vaccine trial (results here and here)

- The Oxford/AstraZeneca COVID-19 vaccine study (results here and here )

These were chosen as they are the two COVID-19 vaccines currently approved and being rolled out across the UK. The HEALTHY STATS public involvement group is part of the NIHR funded HEALTHY STATS research project, which is aiming to improve the information reported from clinical trials on the health benefits of new treatments to patients. Members include public partners between the ages of 20 to 70 of mixed ethnicities and sex.

These are the key insights I learnt from our discussions:

Not everyone understands what vaccine effectiveness means for reducing symptoms of COVID-19 infection

On November 18th 2020 Pfizer/Biontech announced that their vaccine was ‘95% effective’.

Oxford/AstraZeneca released results on 23rd November, reporting that their vaccine was up to ‘90% effective’, but this was if a half dose/full dose vaccine combination was administered. If a full dose/full dose combination was administered, it resulted in 62% effectiveness. The overall effectiveness across both regimens was 70%.

The public involvement group found the three results for the Oxford/AstraZeneca study less straight forward to understand. After explanation they were surprised the lower dose worked better in the trial and hadn’t appreciated this distinction before. In the study the half dose/full dose vaccine had been given by accident to a smaller number of participants (all aged under 55 years old). There were concerns that the overall result might be misleading. But It was felt important to know all three results and have the differences between these numbers clearly highlighted.

These statistics on vaccine effectiveness made it into the headlines of many news stories. In general, the group were impressed with the high numbers reported. However, not everyone fully understood exactly what 95% or 90% effective meant.

So what exactly does ‘95% effective’ mean? If you take two equally large groups of people with diverse characteristics; one group is vaccinated and the other is not: we would expect there to be 95% fewer cases of symptomatic COVID-19 in the vaccinated group. Or you could say, for every 100 cases of COVID-19 discovered in the group that did not have the vaccine, we would expect to see only 5 cases in the ones that did have the vaccine. From the point of view of an individual, the result of being vaccinated will depend on many different factors such as age and whether you have more than one illness.

The discussions highlighted that better communication around what vaccine effectiveness means is required . The public would then better understand how well the vaccines work and how at risk they are from symptomatic COVID-19 illness following vaccination.

Numbers of severe COVID-19 cases and deaths during the trial are important information

The number of severe cases and deaths from COVID-19 and how these differed between the treatment groups in the trials were of key interest to the public involvement group. Concerns around whether any deaths had occurred as a result of the vaccine were also raised. This was following previous media reports of a death occurring in a COVID-19 vaccine trial which was later confirmed to be in the placebo group.

For some it wasn’t clear whether the main headline results on vaccine effectiveness were based on a comparison of severe cases and deaths, or all cases of COVID-19 including mild/moderate cases.

The main reported vaccine effectiveness results (i.e. Pfizer/Biontech 95% and Oxford/AstraZeneca – up to 90%) are the differences in the numbers of any mild/moderate/severe symptomatic COVID-19 illness between the vaccine and placebo groups.

For severe cases and deaths; in the Pfizer/Biontech trial after being vaccinated among 10 cases of severe COVID-19, 9 were in the placebo group and 1 in the vaccine group. In the Oxford/AstraZeneca study 21 days after being vaccinated there were 10 cases hospitalised for COVID-19 and one death; all in the placebo group. No deaths occurred as a result of the vaccine in either study.

This shone a light on the importance of clearly describing exactly what patient outcomes are behind the statistical results.

Uncertainty on whether the vaccines reduce asymptomatic cases of COVID-19 infection

The public involvement group questioned whether the trial results meant the vaccines would also reduce the number of asymptomatic COVID-19 infections. There were positive signals from the Oxford/AstraZeneca study that the vaccine could help reduce asymptomatic COVID-19. However, neither of the trials were set up to be able to confirm whether the vaccines also reduce asymptomatic COVID-19 infection. Further research is currently being conducted to explore this further. This fact should be more widely communicated.

More information on vaccine safety and side-effects is desired

It was equally important for the public involvement group to know the vaccine was both safe and effective. Whilst initial reactions on the vaccines effectiveness were good, there were concerns about the lack of reported information on side effects and how severe these might be.

The public involvement group wanted to see more information on safety communicated publically.

Who have the vaccines been tested on?

The group also wanted to know more about the differences of those given the vaccines in the trials, to know how appropriate the results were for them. For example, Pfizer/Biontech indicated that their trial included ‘42% with diverse backgrounds,’ but knowing the ages and ethnicities of participants in trials is important to the public.

Whilst information on participant characteristics and further safety data is available in scientific papers, the group said further work is required to make this information more digestible and available to the public.

How trustworthy is the data?

Some of the public involvement group felt the main vaccine effectiveness results were almost too good to be true, which lead them to question how much trust we can place in the results of trials from big pharmaceutical companies. In a time when misinformation is unfortunately continuously circulating online, members of the public want reassurance and to know they can trust the results from these clinical trials.

Time and effort must be dedicated by clinical trial researchers to engage with communities and build that trust.

Are ‘interim’ results understood?

Interim analysis is when the trial data are looked at partway through before all participant data has been collected; interim results are therefore subject to change. These results are not typically reported but kept confidential.

Some COVID-19 vaccine trials first reported results from interim analysis, followed by updated results. For example, Pfizer/Biontech first reported interim results of 90% vaccine effectiveness. Just over a week later they reported 95%. Not everyone in the public involvement group noticed this difference, although they were described as interim. Not all understood the meaning of interim analysis.

After clarifying the meaning of interim analysis the group did not mind that the results had been communicated in this form, but wanted to be reassured that they could trust such data.

I can’t help but wonder that opinions might have been quite different had the vaccine effect been much lower in the Pfizer/Biontech final analysis. Such results can magnify concerns around how trustworthy data is.

Should statistics from different trials be compared?

Some members of the public involvement group initially felt the statistics could be compared. This revealed that the differences between the trials aren’t widely appreciated. For example, the two trials included different populations of people: Pfizer/Biontech included participants aged 16yrs+ from USA, Argentina, Brazil, South Africa, Germany and Turkey; Oxford/AstraZeneca included participants aged 18+ from UK and Brazil. Different types of content for the placebo were used: Pfizer/Biontech used a saline solution placebo; Oxford/AstraZeneca used a saline solution or meningitis vaccine. COVID-19 illness was measured for Pfizer/Biontech from 7 days after the 2nd vaccine whereas Oxford/AstraZeneca from 15 days after the 2nd vaccine.

Comparison of results from different trials can be misleading, but this is not fully appreciated and should be better communicated to avoid misunderstandings.

Final thoughts…

Never before have the results of clinical trials been so eagerly awaited and received so much media and public interest. I was personally delighted when the results of these trials were released, showing overwhelming evidence of effectiveness. Alongside valuable insights on what statistics needs to be better communicated to the public, the discussions with the HEALTHY STATS public involvement group revealed building public trust in clinical trials results is a key issue. The current spotlight on clinical trials provides an ideal opportunity to engage with the public to build trust and confidence in clinical trial results.

By Suzie Cro and HEALTHY STATS public involvement group members Ania Henley, Joanna C, Manos Kumar, Paul Hellyer and Yasmin Rahman.