Su Corcoran and Kate Pahl
Recently, we took part in a workshop focused on developing a position paper on “Evaluating the Arts” as part of the AHRC-GCRF PRAXIS project hosted by the University of Leeds. As a result of discussing the various different ways in which traditional methods of programme evaluation could be adapted to take account of the arts, we felt it would be useful to articulate our individual approaches to the use of evaluation in relation to some of the projects that we have been involved with.
As a researcher with experience of working with non-governmental and community-based organisations on monitoring and evaluation (M&E) processes (e.g. Corcoran and Wakia 2013&2016), my (Su’s) approach to evaluation often begins at the practice level. As an academic researcher, I am very aware of the criticism directed towards researchers who parachute into a context, conduct their research, and gain later notoriety through their publications, without ensuring impact – or equal recognition – for the communities, partners, and research assistants who contribute to the research process.
Therefore, the collaboration with Glad’s House as part of our BA-funded project (Re)-engaging street connected young people with education in Mombasa, was developed in order to benefit the organisation. We focused on a research question that we both wanted to answer using methodologies that would help the organisation to work towards achieving their advocacy goals. In this instance, developing qualitative research skills to enable data generation as part of M&E processes that could provide evidence for working with local government to better inform initiatives/responses aimed at street-connected young people. A key observation made by the social work team involved in the project was the benefit of using more creative methods of engaging with young people (Corcoran et al. 2020) and therefore the project had a lasting impact not only on the development of qualitative modes of M&E, but also on the activities utilised on a day-to-day basis. The development of more qualitative and/or ethnographic approaches to monitoring and evaluation are being developed by a number of funders in the international development sector. StreetInvest (2017) are encouraging participative and reflective practice approaches to M&E and Comic Relief funded Retrak in Uganda to explore the use of a combination of qualitative data generation methods that could link into storytelling (Gebeyehu and Endeshaw 2018) – both focused on street-connected young people.
My work focuses on contexts that I do not have experience of living, although I have work experience as an educator and administrator in schools and NGOs in these contexts. Therefore, evaluation can play a key role in helping a cross-sectoral team of researchers, artists and practitioners to envisage frameworks that need to be in place for future projects. Our (Kate and Su) AHRC-GCRF Belonging and Learning network project aimed to pilot the use of arts-based methods to encourage dialogue between young people and policy makers. However, in exploring the effectiveness of encouraging local government officials to either dance or write poetry with street-connected young people in Kenya and the DRC, or of bringing policymakers to an exhibition of visual art created by refugees in Uganda, there were wider questions to evaluate beyond the mode of communication (Ferguson 2020).
We took an ethnographic approach to evaluation in this project, inviting Vicky Ferguson, who has extensive safeguarding knowledge, to observe all aspects of the workshops conducted in each of the three countries. As such, we opened ourselves up to extensive critique in relation to our relationship as academics with the practitioners and artists who were our partners on the project. More importantly, we were able to highlight the complexities that need to be negotiated to ensure that both the artists’ roles as facilitator and equal partner in the research is combined and supported with the practitioner knowledge of our community partners in order to provide safe spaces in which young people are heard. And that these young people understand from the outset, what realistic impact from their participation looks like in practice (Ferguson 2020). In order to do this, how can we bring young people’s voices into the project from the beginning – if not at the proposal writing stage then ensuring that the proposal includes time for young people to be consulted and the project to be co-produced.
In the AHRC-funded Questioning the Form project, we are further developing the ethnographic approach to evaluation, engaging the art form around which the project has been developed as the output of a reflective evaluation of the project. Questioning the form will combine poetry and visual art in the creation of zines with women in Uganda. Adapted in light of the COVID 19 situation, two reflective zines will be produced in addition to the zines made by the women who participate. The first will involve a station on one side of the room that will become a collective zine that the women can contribute to at any stage during the two 5-day workshops. Lisa Damon, who is observing the workshops and speaking with the women and the different collaborators at various stages in the project, will create the second. Lisa’s reflection has already started as she is reflecting on how we are working together to adapt our plans to the current – pandemic – situation. As we are unable to travel, Lisa, and the two of us, will interact with the workshops remotely, which offers an interesting opportunity for thinking through a blueprint for future projects that place autonomy for the projects firmly in the hands of local collaborators and consider the growing need to consider climate change.
However, despite the depth and engagement with the artistic form that creative methods of evaluation provide, we have found that policy-maker audiences are not necessarily receptive to such outputs, even though they carry great weight for the participants involved in their creation. I (Kate) have had extensive experience of co-producing research with young people. Evaluative work is only meaningful to young people if the form it takes makes sense to them and can articulate their concerns (Pahl 2019). Film and poetry is sometimes not seen as ‘evidence’ by policy makers. In Rotherham, the AHRC-funded Making Meaning Differently project I led explored young people’s perceptions of government and involved a group of young people making a film to show to government using shadow puppets. Although their message was powerful, the officials preferred Slide Packs as forms of evidence. The work of the young people was not engaged with. Therefore, as we developed the Rotherham project, we took the decision to co-produce a book that was composed of art work as well as writing by the policy-makers, exploring the nature of knowledge and the different forms that knowledge could take within communities.
So where does our experience lead us in terms of identifying innovative ways of capturing learning and impact, and how we can balance the needs of different stakeholders and compile recommendations for ways forward – particularly in terms of the preference for formal reports and quantitative impact data? We feel that there are a number of key questions that should be considered:
What do young people, community partners and artists want/need from the project?
As much as possible, research projects should be coproduced with the people who will be directly involved with delivering or participating in them. What researchers in the UK feel is important may not necessarily correspond to the practitioners at the local level in the DRC for example. The current scramble to complete proposals for funding calls does not necessarily provide the opportunities for equal collaboration – or the involvement of young people, for example, who are the focus of the project. Therefore, the design and budget for projects should necessarily incorporate space in which these different voices are able to articulate their expectations to shape and co-design the project that is finally delivered – from deciding on the form of the project outputs and research methodologies, to the nature of the evaluation process.
What data is required at the local level? Can the project incorporate this? Would the local level data also satisfy the detail at the funder/policy maker level?
There are multiple levels of knowledge creation within any project – for example, from the young people who participated in the projects above to the funders who require that we report on our impact – and it is just as important to understand what all of these various stakeholders would like the project to achieve. In discussing the evidence that is required at the local level for advocacy etc., their requirements may correspond to the expectations of stakeholders who often have more influence on the form that such evidence needs to take.
Does bringing policy makers into the projects mean that a system of data generation can be codesigned once they experience the value of the arts?
The activities we developed in Rotherham and the DRC showcase two clear instances of outputs involving the voices of young people and of policy makers. For future projects, we aim to develop the format of the Belonging and Learning project to involve policy maker engagement throughout. In so doing we hope to raise the profile of knowledge created using arts-based methods and potentially influence policy-makers. Arts methods open up a space which enables people to communicate in more horizontal ways, and thereby change the space of their encounters to communicate more honestly and fully.
References
Campbell, E., Pahl, K., Pente, E. and Rasool, Z. (2018) Re-Imagining Contested Communities: Connecting Rotherham through research. Bristol: Policy press.
Corcoran, S.L., Awimbo, L.A., Mugwanga, K., I. A. Aluoch (2020) Street-connectedness and education in Kenya: Experiences of formal schooling as rationale for inclusive pedagogies of practice. Prospects
Corcoran, S. & Wakia J. (2013). Evaluating Outcomes: Retrak’s use of the Child Status Index to measure wellbeing of street-connected children. Manchester: Retrak.
Corcoran, S. and J. Wakia (2016). ‘Using child wellbeing assessments to track progress in family reintegration’. Global Social Welfare 3:137-145
Ferguson, V. (2020). External Evaluation of the GCRF/AHRC-funded project – Belonging and learning: Using co-produced arts methodologies to explore youth participation in contexts of conflict in Kenya, Uganda and the Democratic Republic of Congo (DRC). Manchester Metropolitan University.
Gebeyehu, M. & Endeshaw, Y. (2018). Final Evaluation of the “Changing lives of vulnerable children and families from the SNNPR” Project. Manchester: Retrak
Pahl, K. (2019) Recognizing Young People’s Civic Engagement Practices: Rethinking Literacy Ontologies through Co-Production Politics of Literacies. Studies in Social Justice 13(1) pp 20-39
StreetInvest (2017) Street Work and Partnership M&E: Guidelines for regional coordinating partners. Twickenham:StreetInvest