Our confusion about evidence-based policy is making it difficult to achieve

Eighty-years ago doctors would prescribe medicine without any type of rigorous proof that it would improve your health. Thirty-years ago physicians relied more on ‘hunches’ and dated education than modern evidence to determine the best treatments. Of course, that seems as preposterous today as the burning of suspected witches does. “How did that lady know it was going to rain today? Must be a witch…someone get firewood.” Hunches no longer pass for evidence these days. Unless we’re talking about government policy.

Randomized controlled trials and evidence-based medicine have become standard practice in the healthcare field, saving millions of lives as a result. Government policy has the same potential to benefit society. And while ‘evidence-based policy’ has become a pursuit for policymakers, many don’t understand what they’re pursuing.

The term ‘evidence-based policy’ refers to using previous research and the ‘scientific method’ to make government decisions. It is intended to leverage past study findings, along with evaluation and experimentation approaches from fields such as health, economics, and other social sciences to generate proof of the impacts of specific policies or interventions. This is done much less than one would expect.

As policy expert David Halpern, puts it:

“The dirty secret of almost all government is that we don’t know whether we’re doing the right thing. Whether what we’re doing is helping the patient or not helping the patient, or even killing him. People are often running blind, overconfident, and dispensing massive sums of money.” (Apolitical, 2018)

Halpern has spent years studying and working in senior government decision-making which makes his statement both shocking, and true. For twenty years, I have worked with government to help guide decision-making, and I’m embarrassed to admit that it was only five years ago that I learned about evidence-based policy. I was still working to simply get data included in the decision-making process, let alone real evidence.

What’s the difference between data and evidence? It may seem nuanced, however, understanding the difference is critical. It is this lack of understanding that has made so many governments believe they are using evidence when they are not.

Data is raw information that quantifies a situation — hospital bed utilization, student loan repayment rate, jobs created in the last month, etc. Data is integral to evidence-based decision making. Having the numbers alone, however, does not make something evidence-based.

Evidence is generated using an approach intended specifically to do so. This could be done with a formal evaluation or experiment where a study is designed, data is gathered, and the results are interpreted by experts. This approach explains the root cause of the data so that outcomes can be predictably replicated or avoided, thereby generating ‘evidence’.

Here’s an example. Collecting the test scores of students is an example of gathering ‘data’. Observing that they are increasing, forming an opinion, and acting on it isn’t an example of using evidence. It is informed decision-making, but no one knows why test scores are increasing based on just collecting this data. Some type of intentional study would be necessary to determine the root cause (i.e. evidence) leading to improved outcomes. Measuring the difference in the test scores between semesters for two classes, being taught by the same teacher, but using different curriculum might be ‘evidence’ that one curriculum is better than the other.

Why does this matter? Evidence-based policy has the potential to improve society in the same way as evidence-based medicine and drug trials. It allows us to isolate the effect of policies and determine which ones actually improve people’s lives, which will allow us to replicate them to create a bigger impact.

The broad misunderstanding within government of what evidence-based policy is has been a major challenge to its use, but this hasn’t been the only challenge. In my next blog, I’ll describe what I believe is another challenge to practicing evidence-based policy in government — government itself.

Be sure to sign-up here and receive my next blog as soon as it goes live.

About the Author: Mike Davis is CEO at Davis Pier / Advising on social innovation, government transformation, and evidence-based policy