In many contexts, evidence and decision-making are paired with implicit acknowledgement that actions taken can be justified by what we know. There are various paradigms for evidence-based decision-making that are used in different contexts. Clinical guidelines, for example, that are increasingly the basis for providing state-of-practice care, are grounded in a foundation of evidence generally brought together through systematic review. Such comprehensive reviews end with a determination of the weight of evidence in support of a particular proposition and also characterize the uncertainties related to that support. Uncertainty about what is not known inevitably complicates decision-making, and the level of uncertainty needs to be understood in informing judgments about decisions to be taken.  

As a public health example, the World Health Organization released its revised Air Quality Guidelines last week. I participated as a member of the guideline development group for the 2005 update and this most recent one as well. The differences in process between the two updates illustrate the current approach to transparent incorporation of evidence into guideline formulation. 

For the 2005 updates, reviews were written in advance by the group’s members, but not following predefined protocols. The group then met for a week, reviewed and rewrote the chapters on the various pollutants, and set target concentrations for air pollutants through a consensus process. For the new guidelines, the process began with design of systematic reviews, consideration of their findings, and development of a methodology to move from the reviews to numeric values for the guidelines. While transparent, the process brought challenges and was lengthy, lasting five years from the start to the release of the guidelines. The WHO uses the GRADE (Grading of Recommendations, Assessment, Development and Evaluations) system for study evaluation and evidence synthesis. Originally developed for the clinical arena, where the evidence comes largely from randomized clinical trials, its extension to observational epidemiological studies of environmental exposures brought problems and contention as to how to resolve them. Pragmatic solutions were found to these problems and the foundational systematic reviews completed. From the evidence compiled in the reviews, the group identified the range of exposures over which significant adverse effects were identified and used that range as a guide to determining the guideline concentrations.

A similarly transparent approach is used in setting the National Ambient Air Quality Standards (NAAQS) in the United States, although the NAAQS differ somewhat from the WHO guidelines. The deliberate and lengthy course of revising the WHO Guidelines and the NAAQS contrasts with the necessary pace of decision-making for managing the COVID-19 pandemic. The distinction is highlighted by the recent consideration of the need for third shots by the FDA and the CDC. On September 22, the FDA issued an Emergency Use Authorization (EUA) for a third dose of the Pfizer-BioNTech COVID-19 Vaccine. The EUA extended to:

• Individuals 65 years of age and older;
• Individuals 18 through 64 years of age at high risk of severe COVID-19; and 
• Individuals 18 through 64 years of age whose frequent institutional or occupational exposure to SARS-CoV-2 puts them at high risk of serious complications of COVID-19 including severe COVID-19.

On September 23, the CDC’s Advisory Committee on Immunization Practices (ACIP) did not recommend a booster for those at risk because of a high level of exposure through their institutional setting or occupation. The committee voted 9 to 6 against recommending a third shot for this large category, offering a range of pro and con opinions regarding their votes. CDC Director Walensky accepted the two recommendations from the ACIP and added a recommendation to cover those in high risk institutional and occupational settings. In supporting her recommendation, she offered: “As CDC Director, it is my job to recognize where our actions can have the greatest impact. At CDC, we are tasked with analyzing complex, often imperfect data to make concrete recommendations that optimize health. In a pandemic, even with uncertainty, we must take actions that we anticipate will do the greatest good.” In this quotation, she captures the challenge of making decisions under uncertainty, particularly when the evidence hangs in the balance, as exemplified by ACIP’s vote. Her decision appears to have tipped by anticipated public health benefits from moving forward with the third group, not recommended by ACIP. This might be viewed as a “no regrets” decision if there were no risk, or only a very small risk, of unanticipated consequences of a third shot – as might be anticipated from experience to date. The presentations to ACIP provide the relevant evidence and analyses.

Sir Austin Bradford Hill ended his 1965 landmark paper on causal judgment with the still applicable comment: “All scientific work is incomplete – whether it be observational or experimental. All scientific work is liable to be upset or modified by advancing knowledge. That does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time.” Followed by: “Who knows, asked Robert Browning, but the world may end tonight? True, but on available evidence most of us make ready to commute on 8:30 the next day.”

Jonathan Samet, MD, MS
Dean, Colorado School of Public Health

In many contexts, evidence and decision-making are paired with implicit acknowledgement that actions taken can be justified by what we know. There are various paradigms for evidence-based decision-making that are used in different contexts. Clinical guidelines, for example, that are increasingly the basis for providing state-of-practice care, are grounded in a foundation of evidence generally brought together through systematic review. Such comprehensive reviews end with a determination of the weight of evidence in support of a particular proposition and also characterize the uncertainties related to that support. Uncertainty about what is not known inevitably complicates decision-making, and the level of uncertainty needs to be understood in informing judgments about decisions to be taken.  

As a public health example, the World Health Organization released its revised Air Quality Guidelines last week. I participated as a member of the guideline development group for the 2005 update and this most recent one as well. The differences in process between the two updates illustrate the current approach to transparent incorporation of evidence into guideline formulation. 

For the 2005 updates, reviews were written in advance by the group’s members, but not following predefined protocols. The group then met for a week, reviewed and rewrote the chapters on the various pollutants, and set target concentrations for air pollutants through a consensus process. For the new guidelines, the process began with design of systematic reviews, consideration of their findings, and development of a methodology to move from the reviews to numeric values for the guidelines. While transparent, the process brought challenges and was lengthy, lasting five years from the start to the release of the guidelines. The WHO uses the GRADE (Grading of Recommendations, Assessment, Development and Evaluations) system for study evaluation and evidence synthesis. Originally developed for the clinical arena, where the evidence comes largely from randomized clinical trials, its extension to observational epidemiological studies of environmental exposures brought problems and contention as to how to resolve them. Pragmatic solutions were found to these problems and the foundational systematic reviews completed. From the evidence compiled in the reviews, the group identified the range of exposures over which significant adverse effects were identified and used that range as a guide to determining the guideline concentrations.

A similarly transparent approach is used in setting the National Ambient Air Quality Standards (NAAQS) in the United States, although the NAAQS differ somewhat from the WHO guidelines. The deliberate and lengthy course of revising the WHO Guidelines and the NAAQS contrasts with the necessary pace of decision-making for managing the COVID-19 pandemic. The distinction is highlighted by the recent consideration of the need for third shots by the FDA and the CDC. On September 22, the FDA issued an Emergency Use Authorization (EUA) for a third dose of the Pfizer-BioNTech COVID-19 Vaccine. The EUA extended to:

• Individuals 65 years of age and older;
• Individuals 18 through 64 years of age at high risk of severe COVID-19; and 
• Individuals 18 through 64 years of age whose frequent institutional or occupational exposure to SARS-CoV-2 puts them at high risk of serious complications of COVID-19 including severe COVID-19.

On September 23, the CDC’s Advisory Committee on Immunization Practices (ACIP) did not recommend a booster for those at risk because of a high level of exposure through their institutional setting or occupation. The committee voted 9 to 6 against recommending a third shot for this large category, offering a range of pro and con opinions regarding their votes. CDC Director Walensky accepted the two recommendations from the ACIP and added a recommendation to cover those in high risk institutional and occupational settings. In supporting her recommendation, she offered: “As CDC Director, it is my job to recognize where our actions can have the greatest impact. At CDC, we are tasked with analyzing complex, often imperfect data to make concrete recommendations that optimize health. In a pandemic, even with uncertainty, we must take actions that we anticipate will do the greatest good.” In this quotation, she captures the challenge of making decisions under uncertainty, particularly when the evidence hangs in the balance, as exemplified by ACIP’s vote. Her decision appears to have tipped by anticipated public health benefits from moving forward with the third group, not recommended by ACIP. This might be viewed as a “no regrets” decision if there were no risk, or only a very small risk, of unanticipated consequences of a third shot – as might be anticipated from experience to date. The presentations to ACIP provide the relevant evidence and analyses.

Sir Austin Bradford Hill ended his 1965 landmark paper on causal judgment with the still applicable comment: “All scientific work is incomplete – whether it be observational or experimental. All scientific work is liable to be upset or modified by advancing knowledge. That does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time.” Followed by: “Who knows, asked Robert Browning, but the world may end tonight? True, but on available evidence most of us make ready to commute on 8:30 the next day.”

Jonathan Samet, MD, MS
Dean, Colorado School of Public Health