How Well Does COVID Public Policy Align with Science?

The rapid development of effective COVID-19 vaccines is a powerful example of how the business world and the scientific world can successfully work in tandem. But there is a thornier, related question: How well does public policy coevolve with science?
 
In the context of a major existential threat like the COVID pandemic, that means examining how well national, state, and local policies on social distancing, school and business closures, mask wearing, and the like align with the latest scientific findings on the virus.
 
The issue is critical because any disconnect between science and policy could mean a much greater loss of life and economic hardship. Indeed, there has been general skepticism about how well the science and policy communities communicate, and concern about how well policymakers can differentiate well-supported scientific ideas from less-established ones. 

COVID-19 has presented a unique opportunity to study the link between policy and science. “It has been difficult in the past to quantify the impact of science on policymaking, especially in global and fast-moving contexts,” said Dashun Wang, an associate professor of management and organizations at the Northwestern Kellogg School of Management and (by courtesy) industrial engineering and management sciences at Northwestern Engineering, and director of Kellogg’s Center for Science of Science and Innovation (CSSI).
 
To understand the coevolution of science and policy, Wang and colleague Ben Jones, a Kellogg strategy professor, along with CSSI researchers Jian Gao, a Kellogg postdoctoral research fellow, and Yian Yin, a McCormick School of Engineering doctoral student studying IEMS, analyzed tens of thousands of policy documents worldwide. They studied the scientific research those documents drew on, as well as COVID-related documents from intergovernmental organizations like the World Health Organization (WHO). They were interested in how the type and volume of policy documents was influenced by the number of COVID cases in a country, as well as how closely policymaking relied on scientific research.
 
Overall, their study revealed good news: “We found that global policy attention to COVID-19 closely follows the number of cases and draws heavily on the rapidly evolving science,” Wang said. “And COVID policies rely on validated science. So there’s great synchrony between policy attention, the realities of the outbreak, and sound research.”

How policy evolves

To get at links between COVID case rates, policy, and science, the researchers analyzed over 37,000 policy documents across 114 countries—all major economies except mainland China—from the start of 2020 through May 26. This included publications from government agencies, think tanks, and IGOs. They also scrutinized the scientific research cited in the policy documents.
 
They started by comparing policy documents focused specifically on COVID with all other policy documents. They found that the share of COVID policy documents represented within the total policy documents followed a nearly identical trajectory to the number of confirmed cases worldwide. As the number of cases went up, so did the level of policy attention.
 
And the volume of policy documents wasn’t the only measure that followed pandemic trends; the focus of policy shifted, as well. As COVID-19 effects shifted from strictly health-related to socioeconomic concerns—business closures and unemployment, for example—the number and share of policies related to those areas increased. For example, policy documents in January and February 2020 focused mostly on medical and public health issues, but the share of economy-related documents rose starting in mid-March, as governments started contemplating and implementing shutdown orders.
 
Importantly, the team also studied the recency and quality of the scientific research on which policy documents relied.
 
“Do they actually use good science or dubious science?” Wang asked.
 
One concern has been around scientific-article “preprints,” which are papers that have not yet been peer-reviewed. “There has been a flood of COVID research on preprint servers,” Jones said. “Some of these papers have received massive media attention only to be discredited later.”
 
The researchers found that pandemic-related policy documents drew on not only extremely recent science but also work that had been peer-reviewed. For example, nearly 20 percent of papers cited in COVID-related policy documents were published in 2020, about 10 times the rate seen for non-COVID policy documents. And the articles tended to be those published in respected medical-research journals like The Lancet.
 
Moreover, the papers cited by pandemic policy documents were more likely to be those referred to more often by other scientific papers, a typical measure of research quality. Research referenced in policy papers had 40 times more citations, on average, than a broader set of COVID research articles not included in policy documents.
 
“This was important because we know science had a very strong response to COVID-19, as the fast pace of vaccine development suggests,” Wang said. “So it’s good to know that policy kept up.”

IGOs as policy conduits

Last, the researchers examined the role of governments and intergovernmental organizations (IGOs) in promoting pandemic-related policies.
 
The results suggest that the policies and communications of large, well-known IGOs like the WHO are critical in tying cross-border policies together and drawing science into policy.
 
“IGOs, especially the WHO, are a key conduit in absorbing science and then publishing related policy documents,” Jones said. “Then those policy documents are cited worldwide.” As the authors write, national governments “consume science largely indirectly through the IGOs.”
 
Moreover, when more countries and regions follow the same IGO guidelines, the result is a more unified, consistent response. But some countries, including the US, have been “antagonistic” toward large IGOs like the WHO, Wang noted, which can lead to policies less in line with well-documented science. This could potentially mean fragmented implementation and negative health and economic consequences.
 
“In an era of misinformation, some good news is that policy documents emphasize vetted science, including the latest science,” Jones said. “But different countries behave differently, and science is heard unevenly.”
 
That begs an important question: Is there a link between how individual countries and governments use science to inform their COVID-related policy and how well they manage the pandemic?
 
That’s exactly what the team is looking into now, Wang explained. “We’re investigating country-level heterogeneity in leveraging science for policymaking, and what the consequences of that are for critical outcomes like citizens’ health.”