‘Policy based evidence making’
Between the recurring bouts of existential crisis brought on by a combination of the demands of RSA change management, the deteriorating form of West Bromwich Albion and night time flatulence (for which, apparently, the only cure is to give up every single type of food I enjoy eating), I have been thinking about the relationship between evidence and belief.
One prompt was an LRB review of Wilkinson and Pickett’s ‘The Spirit Level’ by David Runciman. David argues that Wilkinson and Pickett overstate the statistical evidence of the damaging effects of inequality on all levels of society. They do this, he argues, because they hope the statistics will relieve them of having to make what is ultimately an ideological claim; namely that inequality is a bad thing. If we are inclined to think this, says Runciman, there is enough evidence out there for us to make our case (and Wilkinson and Pickett assemble the best of it), but trying to prove it with facts alone is not only self defeating but misunderstands how political change works.
The question was also raised by a wonderful little paper recommended to me by my old IPPR colleague Joe Hallgarten. ‘On bullshit in cultural policy practice and research’, is by Dr Eleonora Belfiore from the Centre for Cultural Policy Studies, at the University of Warwick. Dr Belfiore mines Government documents and ministerial speeches on the impact of arts investment on social objectives such as educational attainment and social inclusion. Not only does she find arts ministers making claims which have no basis in evidence but she also reveals how the same ministers occasionally drop their guard and admit the pretence is required of them in order to convince the Treasury or Number Ten to maintain cultural funding. She concludes:
“At the heart of the notion of ‘performance paradox’, thus, is the baffling observation that measures such as the imposition of targets, performancemanagement, evidence-based policy-making, pressures to evaluate the extent to which arts project have the socio-economic impact that policy makers presume they do – or in other words a whole range of measures introduced with the aim to improve transparency and accountability in the public sector – might have resulted, in reality, in more bullshit being produced and injected in public discourses around policies for the cultural sector, and in opaque political messages amounting to little more than doublespeak”.
The point I take from these two essays is that trying to prove arguments in social policy can not only be self defeating, but may involve us in hiding our beliefs behind ‘facts’. I am taking this to heart as I desperately try to finish my annual lecture before I am due to deliver it on Thursday evening.
The speech explores the relationship between new thinking about human nature (derived from behavioural research and neuroscience) and the attempt to close what I have called the ‘social aspiration gap’, enabling people to living more engaged, self reliant and altruistic lives.
The temptation in all this is to overstate the evidence. This is a criticism fairly directed at my piece on brains and ideology in Prospect magazine. Indeed this month’s edition contains a forthright letter from one of the magazine’s own editorial team making this point (is this a first I wonder: an essay so unfortunate that it made the commissioning magazine’s editors turn on each other!).
A theme running through my annual lecture is that we overstate how much control we exercise over our own behaviour and prospects as individuals, and understate not only the importance of, but the capacity we have to influence, our social environment. But I will be sure to make clear that, while there is research to reinforce this belief, there will never be enough to prove it.
Related posts:
- Getting well-being right Will the growing interest in measuring well-being rather than simply...
Comments
10 Comments on ‘Policy based evidence making’
-
karl on
Tue, 27th Oct 2009 1:34 pm
-
Joe Nutt on
Tue, 27th Oct 2009 9:41 pm
-
Matthew Kalman on
Wed, 28th Oct 2009 12:33 pm
-
Katie Boswell on
Wed, 28th Oct 2009 2:44 pm
-
Joe Nutt on
Wed, 28th Oct 2009 5:26 pm
-
Matthew Kalman on
Wed, 28th Oct 2009 8:54 pm
-
Michael on
Fri, 30th Oct 2009 9:08 am
-
matthewtaylor on
Fri, 30th Oct 2009 6:33 pm
-
matthewtaylor on
Fri, 30th Oct 2009 6:42 pm
-
mike barnato on
Tue, 3rd Nov 2009 6:26 pm
Public policy in relation to sport is a good area to explore these thoughts. There are whole load of quotes by ministers over the years claiming sport improves exam results and health and reduces crime see here http://www.cadenceworks.co.uk/news-views/2007/sport-up-for-grabs for thoughts on this from a few years ago. It may or may not, but many people like it and it makes them feel good …. are they good enough reasons for governments to support something? The evaluation of the the Home Office’s Positive Futures project (by Substance in Manchester) shows how it is engagement that is the key and sport engages some, art others. Another old ippr colleague (Jamie Cowling) did a great report called Art for Art’s Sake is relevant too.
I want to believe Wilkinson and Pickett and am sure I read it less critically as a result. Similarly there are some solutions to night time flatulance that I would want to believe in more … could it be related to the WBA form?
That Dr Belfiore or anyone finds the massive disconnect between politically driven measurement and observable reality a “baffling observation” …is what baffles me. As Karl points out, how many bloated Olympic promises of regeneration or lasting legacy have ever materialised? Yet how much power and social value do the games bring to the millions of people who merely spectate. Inequality is just the price one pays for freedom. Worth every penny I’d say.
INCLUDING A ‘DEVIL’S ADVOCATE’ IN ALL RESEARCH?
This suggestion may make me look very stupid… (as I don’t know the ins and outs of commissioning research etc) but….
Why can’t any proposed government-funded research or intervention include at the outset in some little appendix a short informal ‘Devil’s advocate’ (?) comment/prediction – written by someone who is *not* involved in the research and is known to take a substantially *different* (even opposed) approach to the one that’s being funded.
Just a couple of hundred words where that person might say something along the lines of “I don’t think that intervention X will be found to cause Y in the end, as they’ve left out variables Z. Instead we’ll just see A, B, and C occur”.
I think ensuring the inclusion of a such a devil’s advocate comment might force a bit more nuanced thinking on people… perhaps.
At least in hindsight…
And might be a way of getting two bits of research for the price of one! 
(Presuming the Devil’s Advocate would do this for fame, rather than money).
Or maybe it’s all being done already via peer review, literature reviews and whatnot…?
Matt
Reading this blog piece, I couldn’t help but think back to the Michael Sandel lecture on justice at the RSA a few weeks ago (see http://www.matthewtaylorsblog.com/thersa/michael-sandel-and-finding-the-transcendent-moment-in-debate/). Just as Sandel argues that a concept of justice must necessarily engage with the different moral categories that we bring to the world rather than being based soley on ulititarian calculus or maximisation of freedom, so there is an argument that policy must necessarily engage with moral categories rather than being reduced to an evidence-based calculation of whether, say, inequality is statistically a ‘good’ or a ‘bad’ thing.
My personal view is that evidence is undoubtedly important to policymaking, particularly where it challenges our existing beliefs and assumptions. However, if policymakers are clearer about the distinction between ‘facts’ and their own ‘beliefs’, then we will be able to better weigh up policy ideas on the basis of both evidence and moral beliefs. As Sandel stresses, there is inevitably a role for moral categories and beliefs. The question is whether we disguise these moral categories behind so-called facts or whether we are open about our beliefs and about whether or not the evidence fits with our beliefs.
On a related note, an interesting piece of work came out this morning from the Joseph Rowntree Foundation which challenges the beliefs behind welfare to work policies about the social and economic benefits that work delivers to individuals. See http://www.jrf.org.uk/publications/work-worklessness-deprived-neighbourhoods
Matthew K,
Utterly brilliant idea, honestly. I’ve just spent half a day reading US education department research (the product of the meta-analysis of 8 years worth of other people’s research) and the utterly unjustifiable headlines some companies and quangos have made from it are truly shocking. Some of the most blatant selective quotation I’ve ever seen. I don’t even begin to read any kind of research these days, until I have done some “research” on the authors, or commissioning agents.
Hi Joe,
Glad you liked my little idea…
Funnily enough, I was reading recently about a project that was apparently the biggest and most expensive piece of educational research ever undertaken in the US – indeed it was the ‘world’s largest educational experiment’… aiming to “break the cycle of poverty through better education”.
It was called ‘Project Follow Through’, and compared the outcomes of reworking the curricula in a lot of different schools according to a number of different educational models, enabling detailed comparisons.
But when the clear finding was that a rather staid approach to teaching called ‘Direct Instruction’ was clearly the most helpful to schoolchildren, people rapidly lost interest in the evidence. (What I liked was that this rather traditional approach had a better effect on self-esteem than the ‘Self-Esteem’-focused educational models, if I remember correctly. I think the self-esteem approaches actually worsened self-esteem.).
The worst-performing educational models would eventually be the ones that would become US educational policy, under other names!
Here’s a page on it: http://www.uoregon.edu/~adiep/ft/151toc.htm
Perhaps it gets a mention in some of that US research you looked at. I wouldn’t be surprised if the picture is more complicated than the above webpage suggests.
I know what you mean about the need to research the authors themselves before you read their actual research!
Cheers,
Matt
” [the tendency to] overstate statistical evidence in order to avoid having to make what is ultimately an ideological claim”.
Thanks for sharing this observation/insight. The example which comes to mind is how over recent years anti-abortion campaigners haver focussed hard on claims on foetus viabilty when born very prematurely. Also – the contrast between Gordon Brown (long lists of figures, stats etc – as though they will make the case themselves) and President Obama – explains policy decisions by direct reference to the nation’s values and beliefs.
I hope the lecture went very well yesterday.
Thanks Michael. Lecture went fine. I did the right thing in choosing someone (Tim Harford) as a respondent who doesn’t agree with me it made for a lively debate
Thanks Katie. Great comment and thanks also for the fascinating link
Remember that J. M.Keynes said: ‘There is nothing a government hates more than to be well informed; for it makes the process of arriving at decisions much more complicated and difficult.”
Tell me what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!
RSA Pamphlets
RSA Vision
RSA Animate
RSA Comment
- Twitter @RSAMatthew
RSA Bookshop
The RSA Bookshop, in partnership with Blackwell's the UK's leading academic bookseller, offers Fellows and friends an opportunity to purchase books and to explore our speakers’ ideas in greater detail. Visit the RSA Bookshop.