Whatever happened to theory in teacher education?

As initial teacher education increasingly takes place in schools, what becomes of the activities that remain in the university? Studies on this theme have been carried out over the last three years by a team comprising Tony BrownKim Smith and Elaine Hodson, working within the Education Faculty at Manchester Metropolitan University for the Graduate Teacher Programme. This work had led to a recent publication in the Journal of Education for Teaching and one in Teachers and Teaching (read here), which focused on how trainee teachers experience and conceptualise their employment-based course. This month, a paper for the Educational Action Research Journal pursues the theme of theory, but from the point of view of how university tutors working with employment-based trainees reconceptualise their professional role.

Meanwhile two other papers for the Australian journal Mathematics Education Research Journal and the Canadian journal For the Learning of Mathematics have asked the question: what has happened to understandings of school mathematics in employment-based initial teacher education arrangements? These papers were recently discussed by Tony Brown at his recent professorial lecture,

Last month saw the commencement of a larger scale project at MMU and in the north-west region on theory and practice in initial teacher education. This will evaluate most recent changes in teacher education in light of the introduction by the UK Government of School Direct. We are delighted to welcome Dr Harriet Rowley into the team, who will be carrying out much of the research over the coming academic year.  Our best wishes and thanks are sent to Elaine Hodson for a very happy retirement and for her inspiration in getting the project started.

Tony Brown and Kim Smith

‘Interested Amateurs’ and Educational Research

I am currently thinking about Boyte’s (2006) book Everyday Politics, in which he advances a persuasive idea that politics in America has been professionalised creating a rift between politicians and think tank wonks on the one hand, and the citizenry on the other, a rift that needs to be redressed by reconnecting citizens with politics.  I’m considering this idea in relation to contemporary (professionalised) academia, especially in the humanities, to try and understand what it is academics do, developing what type of knowledge relating to whose concerns and so on.  The broader aim is to re-connect knowledge work in education with the public and the mediating institutions of the education system. This aim aligns with an observation made by Prof. Mike Apple (click this link to see Mike speak in Manchester on 4th July) that one reason that the forces of conservative modernisation – the neolibs and neocons – have been so effective is that they speak to the hopes, fears, concerns and dreams of the lay public (read an earlier blog post about this here).  Basically if the Right speak to the public: what are those seeking greater social justice, doing?

This is why I’m interested in a recent exchange on the ESRI blog (read here) between Harry Torrance and Paul Driscoll. Prof. Harry Torrance has vast experience researching education and from this grounding he wrote a post responding to Ben Goldacre’s call for greater RCT’s (Randomised Control Trials) in educational research (read here). A call, as Harry pointed out, that ignored the extensive field of educational research that engages with these very issues.

Paul Driscoll wrote a series of comments responding to Harry’s post, and they continued corresponding by email. (Both have read this attempt to integrate the dialogue.)  Paul is an academic molecular biologist with a PhD in physical sciences but also a soon-to-be chair of the governors of a local school.  He describes himself as an ‘interested amateur’ in educational research, with a particular concern for how his school would measure the impact of the pupil premium,

“I was thinking how does one really measure the ‘impact’ of pupil premium funding (as we are required to do) – we don’t have the properly matched control condition where that funding was not applied. The same will be true in the future since (as of this week) OFSTED will require us to measure the ‘impact’ of strategies applied to assist the ‘more able students’.  Sure, we can compare year groups before and after, but I fundamentally don’t like that because it seems to me that there are too many confounding variables – different students, different teaching staff etc…”

Harry’s comment on this is that it is unreasonable for OFSTED to expect individual schools to evaluate effectiveness in this way and there seems to be some confusion between monitoring expenditure for accountability purposes and undertaking a proper evaluation.

As part of the email exchange Harry suggested a possible quasi-experimental evaluation design for the effects of the pupil premium (PP),

“There is the question of whether or not we have any data from before PP was introduced, but we may be able to compare with similar schools without such funding elsewhere – a form of quasi-experimental design. Properly funded evaluations should be able to collect both qualitative and quantitative data from successive cohorts of schools and students as an innovation is introduced to different cohorts over time. Thus for example T1 might include a sample of schools receiving the intervention and a sample not.  T2 would include a sample of those now with one term’s/year’s experience (T1 + 1), a sample where it’s new (T2), and those not yet included; etc. etc. over time. Proper matching may remain an issue but such panel studies are likely to be as good as it gets when real schools are trying to get on with their ‘day job’ of teaching pupils rather than helping with research studies. Overall, in such evaluations, we would try to understand not just whether something ‘works’, but how it works, why it works, how it is sustained (or not) over time. This is what I meant in the blog when talking about longitudinal qualitative data collection running alongside a trial.”

Both Paul and Harry found common ground on the difficulties of designing such evaluations, with Harry noting that in the busy context of school-based decision-making some evidence is better than no evidence, even if it’s not generated by an RCT.  Similarly Paul wrote,

“And frankly the more I think about it the more I don’t care, since the pupil premium is a ‘political construct’ designed to make certain politicians look good in a system of declining resources. We will just try to the best for all of our students regardless of background.”

It is in regard to these broader issues and the admirable intention to do the ‘best for all of our students regardless of background’ that educational research performs a crucial and integral function. Harry wrote,

“it’s difficult even to try to agree on what these terms [pupil achievement and motivation] mean to different social actors and how we would know them when we see them, if we see them.  How is morale or motivation manifested? Some might argue that this is irrelevant noise and that as long as test scores go up we know things are ‘working’. But then you get into debates about what we are trying to achieve through education – not just test scores surely – the grade inflation of the last 20 years must give us pause for concern there. The quality of the educational encounter is important, as is the pursuit of a wide range of outcomes. The evidence we have from educational research over many years tells us that it is the quality of teacher-pupil interaction, the vitality of what goes on in the classroom, that is important when trying to improve educational standards.

So, because it’s difficult doesn’t mean we shouldn’t try. But it does mean we have to be modest in our claims. The strength of qualitative approaches is that direct, face-to-face questioning and observation of teachers, pupils, parents etc. can give us some purchase on what the above terms mean and how they are operationalized. Thus we can build up a picture of, for example, not just how many teacher-parent discussions have taken place over a period of time (which may or may not have increased with the introduction of PP funding) but also what the content and quality of the discussion is like. Similarly there are well-established ways of collecting and analysing such data, it doesn’t have to be RCTs vs. anecdote. The library shelves groan under the weight of social science research methods texts.”

Paul, ever the scientist, is happy to state what he does and does not know,

“And please let me declare again my general ignorance of research in education. I am only that interested amateur mentioned at the start. I have no axe to grind. But I would welcome more and importantly accessible debate of these issues [the evidence base of public services and policy], which – it seems to me – should fundamentally underpin so much of our political discourse on public policy in all spheres.”

It is the ‘general ignorance’ of the ‘interested amateur’ that I’m interested in. The idea of re-connecting educational knowledge work with the public remains an idea to be developed, with blogs such as this being one such possible vehicle. What an ‘accessible debate of these issues’ would look like is an interesting question.  Maybe ESRI could develop a pamphlet titled ‘the merits of qualitative research’ in five easy points – although I’m guessing not everyone would be happy with that.

I’d like to thank Paul Driscoll for contributing to this post.  The idea was explicitly not to say that educational researchers (Harry) know more about educational research than people with other jobs or interests (Paul) but rather to ask what we as educational researchers need to do to connect thinking about education with those many and varied people governing schools, picking schools for children, or voting in elections.

James Duggan

Summer Institute in Qualitative Research provocations get even more provocative!

The 3rd International Summer Institute of Qualitative Research (SIQR) takes place here at MMU during 22 – 26 July, 2013.

Developed and directed by Prof Maggie MacLure, the Summer Institute is widely acclaimed as a hot house of cutting edge qualitative research, providing an opportunity to learn about the latest in theory and methodology, in dialogue with leading international theorists.

Once again, the roster of keynote speakers is stunning:

  • Elizabeth St. Pierre (University of Georgia) Post Qualitative Research: The Critique and the Coming After
  • Hillevi Lenz Taguchi (Stockholm University) The Master’s Tools Reactivated? What Kinds of (Researcher) Realities do the New and Renewed Turns in Feminist Qualitative Research Produce?
  • Alecia Y Jackson (Appalachian State University) Rhizovocality Revisited
  • Alison Jones and Te Kawehau Hoskins (University of Auckland) Object Lessons: ‘Vital Materiality’ and Teaching
  • Maggie MacLure (ESRI, MMU) ’The First Secret of the Stammerer’: Researching without Representation?
  • Lisa Mazzei (University of Oregon) Posthuman Enactments of Vibrant Data
  • David James (Cardiff University) Recognising misrecognition for Bourdieu and Fraser, and Why It Matters
  • Jessica Ringrose (Institute of Education, University of London) Bodies, Affect and Intensities: A Feminist Deleuzian Mapping Methodology for Qualitative Research
  • Harry Torrance (ESRI, MMU) Concentrating Research Investment: Can Qualitative Research Survive the Move to ‘Big’ Social Science?

Concentrating Research Investment: Can Qualitative Research Survive the Move to ‘Big’ Social Science?

But that’s not all!

As in previous years there will be a workshop strand on Putting Theorists to Work


…this year the series of Provocations/improvisations: Encounters Between Art and Qualitative Research will be more extensive and, hopefully, even more provocative! Taking place on Wednesday 24th July 2013 (12.30 – 5.30pm), this will be a series of collaborative events involving artists, musicians, researchers, and art theorists, organised by Rachel Holmes (MMU), Geoff Bright (MMU) and Kelly Clark/Keefe (Appalachian State University)

It’s a week that shouldn’t be missed

And there’s still time to book!

Standard delegate fee: £345, inc lunch and refreshments

Initial enquiries and registration details: siqr@mmu.ac.uk

Tel +44 (0) 161 247 2010

Welcoming Stephanie Daza

We are delighted to welcome another new colleague to ESRI. Dr. Stephanie Daza joined ESRI as a Research Fellow in June, from the University of Texas-Arlington. Stephanie earned her doctorate in Cultural Studies and Social and Cultural Foundations of Education at The Ohio State University, under the supervision of Patti Lather and Peter Demerath (in the School of Educational Policy and Leadership), and Abril Trigo (in Latin American Cultural Studies). Based on a year of ethnographic fieldwork, her dissertation examined local responses to globalization in Colombian higher education and particularly student responses (e.g. graffiti, strikes, referendums, etc.) to civic education policy and curriculum post Colombia’s new constitution of 1991. Subsequently she has worked at Eastern Michigan University and UT Arlington.

At Ohio State, Stephanie developed and directed an office of New Diversity Initiatives to address the needs of Middle Eastern students post-September 11th. She also helped build Latino/a Studies and Ethnic Studies. She is currently working on a book on STEM culture and grant-science based on six years of research on two National Science Foundation grants aimed at diversifying Science Technology Engineering and Math education and careers. Stephanie has core interests in research methodology and critical ethnography, with articles in, and forthcoming in, Educational Theory, Qualitative Inquiry (read here), International Journal of Qualitative Studies in Education (read here and here), Race Ethnicity and Education (read here and here), and Educational Studies.

Formerly, Stephanie served as the Postcolonial Studies and Education co-chair in the American Educational Research Association; a US Peace Corps Volunteer in Bolivia; and a public school teacher in California and Ohio, USA. On a personal note, Stephanie is the co-parent of Santana, age nine; he has dual citizenship and lives part of the year in Colombia, South America. She is based in Old Chapel 1.9 so do drop by and say hello if you haven’t met Stephanie already.

Harry Torrance

Some holiday reading – Why so negative?

There’s something very precious and contrary in these austere times in having a holiday away. Not mentioning the carbon cost to the planet, when many are suffering greater hardship and others working zero-hour contracts getting to go to a conference in America and spend time traveling around afterwards seems to be the kind of thing that should only be whispered about out of embarrassment, and the fear that someone in government will hear about it and, following the ‘bed room tax’, introduce a ‘day-off subsidy’ for anyone not in the private sector.  Well, in hushed tones, after attending the brilliant ICQI 2013 I had a two-week holiday in Chicago and New York… where I did a lot of reading. I never said I was a happy-go-lucky guy. Anyway, I’m going to write a few posts about some of the reading I did while away, first up:

Why so negative?

During Maggie MacLure’s pre-BERA seminar (read about it here) a critical theorist explained that she felt a great deal of anxiety about analysing or even talking about a video of a group of small children happily dancing.  I was struck by the idea of what is critical educational research that can’t engage with happy children?

I remembered this while reading the excellent David Graeber’s ‘Toward an Anthropological Theory of Value’, in which he says of critical theory:

The idea was always to unmask the hidden structures of power, dominance, and exploitation that lay below even the most mundane and ordinary aspects of daily life. Certainly

Unknownsuch things are there to be found. But if this is all one is looking for, one soon ends up

with a rather jaundiced picture of social reality. The overall effect of reading through this literature is remarkably bleak; one is almost left with a Gnostic feeling of a fallen world, in which every aspect of human life is threaded with violence and domination. Critical theory thus ended up

sabotaging [its] own best intentions, making power and domination so fundamental to the very nature of social reality that it became impossible to image a world without it. Because if you can’t, then criticism rather loses its point. (Graeber 2001: 30)

While on holiday, I also read Andy Merrifield’s (2011: 110-11) ‘Magical Marxism’ – cheers Cassie for the recommendation! – in a section titled ‘The Prison-House of Negativity’, he says,

An issue here is the problem of negativity, a constant bugbear of the Marxist tradition from the very beginning… To a certain extent Marxism’s Hegelian origins explain a Unknown-1lot about why this has been so.  After all, Marx took plenty from Hegel’s idealist thought, a thought that suggested history hinged on ‘dialectical movement’, an immense epic of the mind striving for unity… [In Hegel’s ‘Phenomenology of Spirit (1807)] Unity is the unity of contradiction, of negative force, of looking the negative in the face and living with it. Without contradictions, everything is void, nothingness… For Hegel, in short, world history is dramatized by the darker side of things, through what things aren’t, through denying (cf. the Latin negare), through the predicate not-something.

So, now that I’m back from holiday I’m thinking about whether Hegel is the, or one of the, reasons why happy children are a theoretical blindspot for critical theory?  If so, how have theorists engaged with the negativity of critical theory or equally how can we can develop a critical theory of something rather than not-something?  I’m also interested in the conditions and relationships between academia and the world out there when talking about something rather than not-something.  A critical scholar can sit in his/ her chair and talk about the not-something of neoliberalism but then to write about something, a preferable something, seems entirely different. In all likelihood it requires something to be out there and exist, a tented city of activists or a food co-operative, unless an author is proposing hypothetical or utopian worlds. Then what is the theorist in relation to this something and how do they help bring into existence less of the not-something and more of the something? There is the danger, as David Graeber describes, of becoming ‘a theorist of a non-existent social movement.’

If anyone has any answers or ideas then please let me know…

Youth Work and the ‘Military Ethos’

Given the widespread culture of support for the Armed Forces after their engagements in Iraq and Afghanistan it may seem difficult to discuss the question of  ‘military ethos’ in schools and also youth work, which has been placed strongly on the agenda by  Michael Gove as Secretary of State for Education. Nevertheless, it is necessary that we should raise questions and debate about this from the perspective of open and democratic perspective in youth work.

The ambition to promote a ‘military ethos’ is currently presented thus on the DFE website (link):

Our ambition is for pupils to use the benefits of a military ethos, such as self-discipline and teamwork, to achieve an excellent education which will help them shape their own futures.

Promoting military ethos in schools helps foster confidence, self-discipline and self-esteem whilst developing teamwork and leadership skills. Past experience from both the military and education sector has demonstrated how these core values help pupils to reach their academic potential and become well-rounded and accomplished adults fully prepared for life beyond school.

We are already working to bring military ethos into our education system to help raise standards and tackle issues such as behaviour. This includes:

  • Expansion of the school-based cadets to create around 100 more units by 2015.
  • Delivering the Troops to Teachers programme, which aims to increase the number of Service Leavers making the transition to teaching.
  • Promoting alternative provision with a military ethos.
  • Exploring how academies and Free Schools can use their freedoms to foster a military ethos and raise standards.

The DFE is working with key charities and CICs which are enabling this work, and, as so often currently, the networks here are text book examples of what Stephen Ball has called the new heterarchies, linking philanthropy, privatisation and peripheralisation.

So, Uppingham School links with the newly constituted Havelock Academy in Grimsby  to develop a Combined Cadet Corps, in the tradition of ‘noblesse oblige’ and Public School influence which informed the origins of youth work.   The three companies with which the DFE is working – Challenger Troop, SkillForce and Commando Joes – themselves – also represent, in their governing bodies, the alliances which are driving so much educational and social policy.  Challenger Troop emerged from Kent voluntary youth services and has a strong base in the Cadets. Its partnerships are with the Police and Fire Service, as well as Housing Associations and Forestry.   Skillforce, based in Failsworth North Manchester and operating nationally, emerged from the army as part of their contribution to civil society.  The process of moving Skillforce out of the Army and establishing its independent governance has been supported by members of the Skillforce Board  who have associations with many establishment bodies such as the Church of England Southwell Minster, the Woodland Trust, and the Racial Justice Committeee of the Baptist Union. Others are the charitable representatives of finance companies JP Morgan and Price Waterhouse Cooper.  The last of the triad funded by the DFE – Commando Joes – is an American company who have not even bothered to change their website for the British market. Alongside its offer of products for primary and secondary schools, sports clubs and birthday parties, Commando Joe’s website is still proudly emblazoned with the logo ‘No Child Left Behind’, the US policy equivalent of Every Child Matters.  With the possible difference that the programmes offered by these organisations may have more emphasis on structure and discipline than much informal education in youth work, they offer much that is important and familiar as ‘outcomes’ of youth and community work processes: confidence, self-discipline, self-esteem, development of team work and leadership skills, inclusion, fun and adventure. What’s not to like?

Support for such strategies has also come from across the political spectrum and has included Labour’s Stephen Twigg as well as David Cameron’s one-time  favourite ‘think tank’ Res Publica.  Res Publica’s advocacy of ‘service schools’ is based on a view of the moral degeneracy of poverty-stricken urban areas. ‘They (service schools) would challenge the cultural and moral outlook of those currently engulfed by hopelessness and cynicism.’  Ex-service personnel can, they argue, ‘act as excellent role models for young people’ whilst cadet experience brings ‘a sense of responsibility and citizenship.’  Through the £15 million grant to the Charity Skillforce ‘an extra 100 ex-service personnel are already making a valuable contribution as mentors for young people in challenging schools and communities across England.’

Unfortunately, without in any way wishing to question the value of individual supportive relationships formed through such work both for young men and for the ex-servicemen,  the discourse of moral elevation and virtue associated with the military can not be sustained in the face of evidence concerning the actual mixed experience of military life.  The organisation Forceswatch gives a full account of and rationale for the critique of the ‘military ethos’ but, even without taking an outright pacifist stance or opposing particular mlitary deployments it is possible to argue that the classroom and youth projects are not war zones and the methods of work in each space should properly differ if we are to retain a sense of education as part of civil society and a space of democracy.

It is sadly the case also that evidence of the ‘morality’ supposedly brought to bear by ex-military personnel can only be put into question by  the levels of rape (one a week) and sexual assault within the services.  High level of violent crime, including domestic assault, as well as homelessness, alcoholism and drug abuse are a matter of record. Dr Deirdre McManus’s research  (published in The Lancet in March 2013 – read here) showed that those who had served in the army were far more likely to commit violent crimes than those who had not, and those who had seen active service in Afghanistan and Iraq were 53% more likely to have committed violent crimes than non-combatants.

It seems likely that the encroachment of the military into civil society is in part related to the changing practice of actual combat and the growth of warfare using drones.  The need to redefine a  role for the army in this context has emerged also in the period since 9/11  and has been analysed by Vron Ware in ‘Military Migrants. Fighting for Your Country.’  In 1998, there was a chronic shortage of troops and only 1% of the British Army came from an ethnic minority. By 2008 the figure was 9% but, Ware demonstrates, 2/3 of these joined as Commonwealth citizens as a result of the need ‘to scour the remnants of the British Empire to find the labour force needed by the army in Afghanistan.  Soldiers from Fiji, Nepal, Gambia and Ghana and the Caribbean were to be found in Army literacy classes alongside soldiers from Dewsbury and Sunderland.  The ‘Community Covenant’ through which the Army is developing its role in civil society, through youth work projects as much as anywhere, may be above all understood as a response to the difficulty the Army has experienced in recruitment, as well as a major contributor to a re-emergent control culture in the inner urban areas.  Ware quotes a significant Army General as saying ‘If you can’t run an Army without migrants, you’re in trouble’ and the Community Covenant is a response to this trouble.

Democratic education, including youth work and informal education, needs a clear response to this network of initiatives concerned with ‘military ethos.’  This response will include an engagement with ideas of international voluntary service as a completely different practice from military service. ‘Service civile’ was introduced in many European countries after the Second World War as a peace-making alternative national military service.  We need to emphasise global connections in our practice and to strengthen emphasis on the disciplines and virtues involved in co-operation. Not only obedience but also dissent requires character, organisation and discipline, but dissident associations which can offer alternatives to the present denigration and abandonment of young people  are not likely to have a ‘military ethos.’ Militarised culture has many attractions, but adventure challenge and team-building have long been part of alternative co-operative education traditions too. On July 4th, the Youth and Community work team at MMU will be supporting a Conference organised by the Co-operative College and MMU  to discuss Co-operative Education against the Crises. It is evidently completely untrue that There is No Alternative to the ‘military ethos’ currently receiving cross-Party support.

Janet Batsleer 

Building Evidence into Education – why not look at the evidence?

Ben Goldacre’s recent paper ‘Building Evidence into Education’ has attracted a good deal of attention and debate (see for example here and here).

In it he argues that more experimental research should be undertaken in education, specifically randomized controlled trials (RCTs).  By implication he criticises educational researchers for not doing this already, and indeed at points in the paper he states that some researchers actively resist such developments (though he provides no examples).

Let’s be clear, much of what Goldacre states makes good sense. He reviews the strengths of RCTs; he argues that they should be used more widely in educational research; he notes the complementary strengths of qualitative research; and he argues that much more time and money should be devoted to producing and disseminating high quality research in education – creating an ‘information architecture’ as he puts it.

Why then do I find the paper so frustrating?  First, because Goldacre does not recognise or acknowledge the fact that educational researchers have been debating these issues for many years (along with most other social scientists). And secondly because the paper ends, as all these sorts of interventions tend to do, with a disciplinary ‘land grab’ for resources.  He concludes ‘We need academics with quantitative research skills from outside academic education departments – economists, demographers, and more, to come in and share their skills…’.  Oh yes, the economists, thank goodness for the economists, who have been so successful in modelling and developing our economy recently. Their RCTs have really helped with that.

In his own words, Goldacre’s paper is a ‘call to arms’. He sets up a rhetorical binary between educational research(ers) – ignorant, incompetent, uninterested in what might improve education – and proponents of randomized controlled trials – knowledgeable, skilled, only looking to identify what’s in the best interests of children.  While ostensibly criticising politicians (Mr. Gove?) for foisting too many untried and untested schemes on education, he plays to the same trope of positioning educators as the ‘enemies of promise’.

Large parts of the paper draw on examples from Medicine.  But given the ostensible focus on Education, might it not have been more useful to look at how these issues have been addressed in educational research over the years? The last time the issue was raised in the UK was probably David Hargreaves’s speech to the TTA in 1996 (Hargreaves 1996). This led (albeit indirectly) to a large programme of research initiated under the Teaching and Learning Research Programme.  The programme featured many mixed methods research designs, including some experimental designs (Torrance 2008). It was led by Andrew Pollard (cf. Pollard 2007), one of the National Curriculum expert panel so recently ignored by Mr. Gove.

However debate in the field long predates this most recent manifestation. Campbell and Stanley’s (1963) classic contribution on ‘Experimental and Quasi-Experimental Designs’ reviews the problems and possibilities of developing RCTs in education in far more detail than Goldacre, noting especially “the intransigence of the environment…that is the experimenter’s lack of complete control”.  In a text advocating experimental design, they nevertheless review threats to internal and external validity at great length and highlight the difficulties of running RCTs properly and effectively. In turn they acknowledge McCall’s (1923) ‘How to Experiment in Education’ and note that there have been regular periods of RCT advocacy and RCT disillusionment in educational research as the clear cut results that RCTs promise have been unforthcoming.

And here’s the rub. The answers to questions of public policy and educational evaluation are often not very clear (nor indeed are the questions sometimes). More circumspect proponents of experimental methods than Goldacre, acknowledge that in order for a causal relationship to be established, even within the narrow terms of an RCT, very specific questions have to be asked. In a collection of papers produced from a conference specifically convened to promote “Randomized Trials in Education Research”, Judith Gueron (2002) argues that while “random assignment . . . offers unique power in answering the ‘Does it make a difference?’ question” (p. 15), it is also the case that “[t]he key in large-scale projects is to answer a few questions well” (p. 40). In the same volume Thomas Cook and Monique Payne (2002) agree that

most randomized experiments test the influence of only a small subset of potential causes of an outcome, and often only one. . . . even at their most comprehensive, experiments can responsibly test only a modest number of the possible interactions between treatments. So, experiments are best when a causal question involves few variables [and] is sharply focused. (p. 152)

Thus RCTs can be very good at answering very specific questions. What they cannot do is produce the questions in the first place: that depends on much prior, often qualitative, investigation, not to mention value judgments about what is significant in the qualitative data and what is the nature of the problem to be addressed by a particular program intervention. Nor can RCTs provide an explanation of why something has happened. That will depend on much prior investigation and, if possible, parallel qualitative investigation of the phenomenon under study, to inform a developing analysis of what the researchers think may be happening.

Much of Goldacre’s paper is devoted to what RCT’s have achieved in medicine. There is little acknowledgement of the differences between medical and educational research. There is virtually no reference to the long history of RCTs in education (i.e. the actual evidence in this field) and how often they result in ‘no significant difference’ being reported between control and experimental groups, even when problems of design and conduct of RCTs have been overcome (or, perhaps, because they haven’t). Goldacre states that ‘there have been huge numbers of trials in education in other countries, such as the US’ (again, by implication, castigating educational researchers in the UK), but says nothing about the lack of definitive results. In fact recent findings from the United States have been disappointing. Viadero, Education Week, 1 April 2009, reports: ‘Like a steady drip from a leaky faucet, the experimental studies being released this school year by the federal Institute of Education Sciences are mostly producing the same results: “No effects,” “No effects,” “No effects”.

We should not be surprised. It was precisely the confounding problems of diverse implementation and interaction effects that produced so many “no significant difference” results in the 1960s in the context of the first wave of early childhood intervention and curriculum evaluation studies. Reflections on such results prompted the development and use of qualitative methods in evaluation studies in the1970s and 1980s. Of course it might still be argued that it is important to know that something doesn’t work.  It can also be argued that this is how knowledge advances in science – especially the natural sciences – the accumulation of many negative results before something significant appears to emerge. But Goldacre’s paper makes no reference to such complications – it simply assumes that RCTs will prove what does work, in a very straightforward manner.

Furthermore, the paper assumes that educational researchers are ignorant of RCTS, but as we have seen, this is not the case. Quite the reverse, educational researchers know all too well the pitfalls as well as the possibilities of RCTs, and are appropriately cautious about what they can achieve. While it might still be argued that undertaking more RCTs will benefit education, it cannot be argued, as Goldacre does in his opening paragraph, that this will provide ‘better evidence about what works best’. RCTs simply don’t provide that level of certainty.

Nor are even positive results easily generalised and disseminated to other contexts. Without a reasonable understanding of why particular outcomes have occurred, along with identifying the range of unintended consequences that will almost inevitably accompany an innovation, it is very difficult to generalize outcomes and implement the innovation with any degree of success elsewhere. A good example of this is provided by California’s attempt to implement smaller class sizes off the back of the apparent success of the Tennessee “STAR” evaluation. The Tennessee experiment compared the effects of smaller class size on student achievement, but worked with a sample of schools.  California attempted statewide implementation, producing  more problems than they solved by creating teacher shortages, especially in poorer neighbourhoods in the state. There simply weren’t enough well-qualified teachers available to reduce class size statewide, and those that were tended to move to schools in richer neighbourhoods when more jobs in such schools became available (see Grissmer, Subotnik, & Orland, 2009).

RCTs might provide more evidence, different evidence, and, if properly funded and undertaken in the context of parallel, large scale, longitudinal, qualitative studies,  ‘better’ evidence of what works and why, for different groups in different contexts.  We certainly need more and better research. But ultimately this must be understood as providing a better resource for collaborative decision-making between researchers, teachers, students, parents and local authorities or clusters of schools.  It cannot define what ‘works best’.  There is no such thing in social action, across time, place and differing circumstances. To pretend otherwise is to assert the primacy of one particular research method over the provision of a wide range of different sorts of evidence to inform debate.

Replacing a system currently at the mercy of political whim, with a system driven by a narrow version of science, isn’t going to improve matters.  Let’s produce better evidence by all means, but we have to be appropriately modest about what research can achieve, and research has to develop in tandem with developing better forms of community engagement with our schools.

Prof. Harry Torrance


Campbell D. and Stanley J. (1963) ‘Experimental and Quasi-experimental Designs for Research on Teaching’ in Gage N. (Ed) Handbook of Research on Teaching Houghton Mifflin, Boston.

Cook, T., & Payne, M. (2002) ‘Objecting to the objections to using random assignment in educational research’ in F. Mosteller & R. Boruch (Eds.), Evidence matters: Randomized trials in education research (pp. 150–178). Washington, DC: Brookings Institution Press.

Goldacre B. (2013) Building Evidence into Education Department for Education, London, Available at: https://www.gov.uk/government/news/building-evidence-into-education

Grissmer, D., Subotnik, R., & Orland, M. (2009). A Guide to incorporating multiple methods in randomized controlled trials to  assess intervention effects. Available at http://www.apa.org/ed/schools/cpse/activities/mixed-methods.aspx

Gueron, J. (2002) ‘The politics of random assignment: Implementing studies and affecting policy’ in F. Mosteller & R. Boruch (Eds.), Evidence matters: randomized trials in education research (pp. 15–49). Washington, DC: Brookings Institution Press.

Hargreaves, D. (1996). Teaching as a research-based profession. Teacher Training Agency 1996 Annual Lecture. London: Teacher Training Agency.

McCall W. (1923) How to Experiment in Education, New York, MacMillan

Pollard, A. (2007) The UK’s Teaching and Learning Research Programme:  findings and significance British Educational Research Journal 33, 5, 639-646

Torrance, H. (2008). Overview of ESRC research in education: A consultancy commissioned by ESRC: Final report. Available at http://www.sfre.ac.uk/uk/

Viadero, D. (2009, April 1). “No effects” studies raising eyebrows. Education Week. Available at http://www.projectcriss.com/newslinks/Research/MPR_EdWk–NoEffectsArticle.pdf