Data-driven decisions: The case for randomised policy trials

In medicine, randomised trials are commonly used for evaluating effectiveness. When a new pharmaceutical is being tested, half the recipients will get the true treatment, while half will get a placebo. By tossing a coin to decide whether a person gets the treatment or the placebo, we can be confident that any observed differences are due to the real effect of the drug.

Increasingly, randomised trials are being used by governments and businesses too. Randomised trials of policing strategies have shown that hot spots policing reduces crime. A randomised trial found that when people in India were given a financial incentive to get their licence earlier, they were more likely to bribe the tester.  A randomised trial in Mexico found that road upgrades boost property prices and reduce poverty. A randomised trial with airline pilots found that providing feedback on fuel use led captains to be more economical, saving the airline a million litres of fuel.

Yet by comparison with health, the uptake of randomised trials in social sciences remains modest. From the 1990s to the 2020s, the number of randomised trials in health has exploded from 10,000 to almost 250,000. Yet over the same period, the number of randomised trials in the social sciences has risen from a few thousand to less than 20,000. For every randomised trial in the social sciences, there are around ten randomised trials in health.

This is all the more startling given the breadth of the social sciences, covering education, crime, employment, homelessness and political engagement. In budgetary terms, governments spend much more on those areas than on health alone. Yet in terms of randomised trials, health remains far further ahead.

In Australia, a study from the think tank CEDA examined a sample of 20 Australian Government programs conducted between 2015 and 2022. The programs had a total expenditure of over $200 billion. CEDA found that 95 per cent were not properly evaluated. CEDA’s analysis of analysis of state and territory government evaluations reported similar results. Across the board, CEDA estimates that fewer than 1.5 per cent of Australian government evaluations use a randomised design.

The relatively small number of randomised trials of social programs is particularly troubling given what the evidence tells us about the programs that are rigorously evaluated. In health, only one in ten drugs that look promising in the laboratory make it through Phase I, II and III clinical trials and onto the market. In education, an analysis of randomised trials commissioned by the US Department of Education’s Institute of Education Sciences found that only one in ten produced positive effects. Google estimates that just one in five of their randomised trials help them improve the product.

This suggests that the best approach in business and government is what US President Franklin D. Roosevelt once called ‘bold, persistent experimentation’. If many promising policies do not work as well as intended, then rigorous evaluation is essential to building a cycle of continuous improvement. Rigorous evaluation guarantees that government policies in a decade’s time will be more effective than they are today. A failure to evaluate runs the risk that we will unwittingly repeat our mistakes. Evaluation puts us in a virtuous feedback loop. Without it, we can end up in a doom loop.

How can governments and companies encourage more rigorous evaluation? There are five approaches that can promote more high-quality evaluations, especially randomised trials.

First, encourage curiosity. Employees quickly come to understand the culture of an organisation. When managers make clear that they value new insights, they give permission for everyone in the organisation to question accepted wisdom and gather better evidence, an approach famously dubbed ‘Test-Learn-Adapt’.

Second, aim for simplicity. People charged with sending out letters, emails or text messages should have the functionality to send two versions, so they can continuously improve the language and messaging of their correspondence. This kind of A/B testing has been standard for market research companies for decades, yet remains rare elsewhere. Another initiative is grant rounds to fund low-cost randomised trials. In 2024, the Paul Ramsay Foundation, Australia’s largest charitable foundation, issued a call for proposals for seven projects of up to $300,000 to be randomly evaluated.

Third, subject trials to ethical scrutiny. This isn’t just the right thing to do; it’s also important for creating an environment in which further trials can be conducted. Ethical scrutiny ensures that the interests of vulnerable people are considered, and that the trial can be expected to improve overall wellbeing.

Fourth, create institutions that promote high-quality evaluation. In 2023, the Australian Government established the Australian Centre for Evaluation. Located within Treasury, the centre has a budget of around $2 million per year, and a staff of around a dozen people. Its mandate is to ‘put evaluation evidence at the heart of policy design and decision-making’. The main goal of the centre is to work collaboratively with government departments to conduct rigorous evaluations, especially randomised trials.

Fifth, think internationally. A few years ago, when researching my book Randomistas, I met with a kidney health researcher whose work involved running large-scale randomised trials. He told me that he no longer worked on single-country trials. Multi-country trials, he told me, provided an inbuilt replication function, and greater assurance that interventions worked across people of different ethnicities. In policymaking, Australia could collaborate with other advanced English-speaking democracies to create Living Evidence Reviews – research syntheses on key topics such as homelessness, job training or policing.

Randomised trials embody a spirit that is at once modest and scientific, accountable and democratic. By acknowledging that some policies might not achieve their goals, we recognise that all of us are fallible. And by rigorously testing what works, we put ourselves on a cycle of continuous improvement. Just as your doctor today has better treatments available than she did a decade ago, programs in education and employment should be more effective than they were a decade ago. Randomised trials can shape better policies, one coin toss at a time.

Originally published in The Canberra Times on 21 October 2024.