First do no harm: is it based on evidence?

Policy is not just a “course of action adopted by an organisation” as a dictionary might suggest. It encompasses, especially in the public sector, a whole series of activities from: conceptualising an idea; to research and consultation; drafting a law or program of action and implementing it; and a process of review and evaluation – hopefully with a view to improving it as needed.

At each stage of this process, the individual people involved are fallible humans just like us, with varying preconceptions, prejudices, ideologies and beliefs; including perspectives based on gender, class, colour and ability. Our perceptions are influenced on how and where we were raised and educated, what life experiences we have had since then, and what efforts we have (or have not) made to identify and address them. Individual perspectives can be compounded by aspects of ‘group-think’ where shared dominant views can go unchallenged, and where party politics or ideals may guide our thinking.

One approach that attempts to address the limitations of our human imperfections is to use “evidence-based policymaking” – a concept (or catchphrase) very popular with a number of governments these days. The idea is to use “scientifically rigorous” studies to analyse problems and identify potential solutions, and to align policy as closely as possible with the needs identified by looking at this evidence.

The concept originated in the medical profession, where scientific studies are well established and need to be rigorous and professional given the risks involved with, for example, introducing new drugs.  The “double blind randomized control trial” is perhaps the most famous of these techniques.

Techniques like this get beyond some of the common mistakes of us fallible humans, such as projecting the personal onto the universal: just because I think, or feel, or have a hunch, that the world is flat (or this new drug is great), does not mean it is true.

Another common mistake is failing to understand that correlation does not prove causation. Just because two patterns can be observed in a group, does not mean one is the cause of the other. There are literally thousands of examples of un-related phenomena that seem to correlate.

Bringing the best available research to each stage of the policy making process should, therefore,  enable us to make the best possible decisions, right?

It is hard to disagree that evidence-based policy-making makes good sense.  The reality, however, as always, is a little more complex.

Scientifically rigorous research can be complex and difficult for non experts to understand. Elected policy-makers (our parliamentarians) are rarely voted for based on their academic levels of intellect, and nor should they be. Permanent civil servants cannot be specialists in every subject area. All of us might, in practice, misinterpret evidence, or selectively filter and use it based on our own preconceptions.

Scientific research often tends to focus on numbers – quantitative research – often within a very narrow and specific context. But not all subjects lend themselves to being easily counted. And when numerical analysis is appropriate, it can be argued that even the double-blind, randomized, placebo-controlled trial is not as free from bias as we would like to think.

Research funding does not exist in a vacuum, and many factors influence what funding is available for what topics at any given time. Funders may be more interested in whether research fuels economic growth, or a university’s research rating, than furthering science or promoting intelligent public policy.  It is quite possible, therefore, that rigorous research relevant to the policy area under consideration may not yet exist.

When we commission new research, to address gaps, care needs to be taken not to frame the research questions and objectives in biased ways. Research that is intended simply to back up a preconceived idea is not helpful, but exists. This is sometimes called “policy-based evidence“.

Available evidence about an issue does not always point to an immediate or easy solution to that problem: evidence does not necessarily tell us what to do next. And where solid evidence does already exist, the conclusions reached by the research processes may be politically unpalatable, or highly controversial. (The role of the evidence base in UK drugs policy is an obvious case in point.)

There are a host of other difficulties in making a simple sounding process a reality, including limitations of the policy-making process itself.

In a nutshell, evidence-based policy making is an imperfect ideal. It is rarely, if ever, fully implemented throughout the whole process; and even if it was, it would still not yield complete information.

So should we give up and leave well researched evidence out of the equation altogether? Of course not. But we also need to be realistic and acknowledge that multiple sources of information can, and should, influence policy-makers; and that all sources are imperfect and ultimately influenced by subjective, fallible people. We all need to begin by knowing ourselves.

Evidence-based, or evidenced-referenced, (or “evidence-infused” or “evidence-influenced”) policy is still a reasonable goal in making the best decisions we can. At the very least, if we are to stay true to our objective of helping, or at least doing no harm, the messy policy-making process could or should at a minimum:

  • attempt to compile and review all available relevant evidence
  • include and fund independent analysis of research if needed (because of quantity / complexity)
  • commission and fund appropriate additional research if significant gaps exist
  • never just ignore relevant existing evidence
  • avoid ‘cherry picking’ convenient studies that reinforce preconceptions
  • work hard to avoid “policy-based evidence”
  • engage with complexity and unintended consequences, especially in monitoring and evaluation
  • recognise the power dynamics between all groups involved (and not involved), and include consideration through lenses of gender, class, colour and ability
  • consider especially the views of those who will be most affected by the policy.

The last three points are especially important. Evidence should build on the principles of inclusive participation, agency and voice already outlined, not work against them.

Further reading:

Stages of a Bill on the Scottish Parliament website.

Everything you ever wanted to know about the theory of policy-making, and perhaps a few things you didn’t, can be found over at the website of Paul Cairney at Stirling University, especially the technical but digestible 1000 words blog posts.  We’ve referenced enough posts above that we should also probably plug the book, though we must confess to not having read it (yet!).

Marston, Greg, and Rob Watts,. “Tampering With the Evidence: A Critical Appraisal of Evidence-Based Policy-Making” Australian Review of Public Affairs, 2003. (pdf)