Data as Medicine
4 core practices of decolonising evaluation
“[W]hile evaluation reports largely tell stories of success, on the ground there is minimal change, communities remain impoverished, interventions cause harm to the environment, and evaluation allows that to happen.” - Bagele Chilisa & Nicole R Bowman
Evaluation has the power to transform ideas into impactful projects. When conducted with cultural competence, and designed with and by the communities impacted, evaluation can help guide project managers and funders into developing and funding projects, which create real transformative change.
However, so often evaluation works as a means to market a project rather than provide opportunities to grow and learn from them. Evaluation criteria is shaped by Western values and measurements of success, and evaluation methods often fail to engage with the ways in which the project developers and participants understand and perceive themselves and the world around them.
How can we change this? We must decolonise evaluation practice.
Decolonising evaluation is a growing field of research and practice across the world, and this publication aims to show what, where and how that works in practise. We’ll be sharing case studies, interviews with practitioners, summaries of webinars, and some international context and resources to inspire and guide you.
While we can’t cover every aspect of decolonising evaluation in a short blog, we suggest starting by taking into consideration four core practices to transform and grow your evaluation process: cultural safety, epistemic justice, the politics of translation, and Indigenous data governance and sovereignty.
Cultural Safety
Observing that Maori communities often did not face adequate care in Aotearoa / New Zealand’s medical system, Irihapeti Ramsden and other Maori nurses pushed for institutional change, asking that all nurses be trained in Kawa Whakaruruhau or ‘cultural safety’.
As Ruth de Souza defines it, cultural safety “demands intentional self and organisational critique to look at how our behaviours, opinions and actions as people who work within institutions can negatively affect the cultural identity and wellbeing of the people we work with. Cultural safety offers arts, cultural and community workers the opportunity to examine how unquestioned ways of doing things can be barriers for groups to engage with us. Individuals and organisations can also use the idea to reflect on their work and address imbalances of power through building equal partnerships.”
Evaluators who are not from the communities of the project they are evaluating, or funders who are demanding evaluation could build in cultural safety into all of their work.
The First Nations Cultural Safety Framework
The Australian Evaluation Society has developed their own ‘First Nations Cultural Safety Framework’ - a collaborative process that engaged with and was shaped by the AES Indigenous Culture and Diversity Committee.
Developing on their principles, the framework provides evaluators and commissioning bodies to explore a series of questions about themselves and the evaluation process, centring that cultural safety is not simply on a check-box activity, but an on-going embodied process.
“Cultural safety is a life-long journey for everyone involved. Once people step into the journey, it is about staying on the journey, not arrival at a pre-defined destination… this occurs through a recursive and ongoing commitment of observing, listening, reflecting, discussing, acting and learning.” (p13)
Epistemic Justice
When thinking about evaluation processes, it’s always important to ask “whose knowledge systems are being valued here?” Within evaluation systems, these are expressed through criteria of success, communication methods, and predefined concepts of what actually constitutes ‘knowledge’. Within academia, this has been explored through the concept of ‘epistemic injustice’ - which explores how identity and systems shape inequity in how knowledge is defined and valued.
It is important to recognise that evaluation is not a Western concept, and that all people and all cultures have had ways of reviewing, reflecting and learning from their actions and activities. How that is expressed and practised, however, can differ. If the body commissioning the evaluation or the evaluator is external to the community that is being supported, it is important that they explore the ways in which that community might already be practising and support an evaluation process that is shaped and defined by that community.
The danger in ignoring those systems of knowledge is that the resulting evaluation is false because the language and criteria used within the evaluation process do not align with the ways in which project participants perceive the world around them. Thus, communities can begin to see evaluation processes as surveillance, where the process works to police their project rather than support it in achieving the goals the participants actually need from it.
Visioning B.E.A.R. Circle Intertribal Coalition Inc
“Initially, I was not a proponent of evaluation because I felt like, oh, this is the way that the government just figures out ways to not support people’s work… What I needed was a new language in which to talk about evaluation because there is something about a historical context that is important around evaluation. I think it needs to be rooted in a history… and our evaluation processes need to reflect that.
We needed a circle process. We needed a non-hierarchical model of storytelling in which to capture the work. And we needed to address the impact that the dominant culture has had on our communities and the damages done and even for us to forget some of the things we were really great at…”
After finding that traditional models only advanced the harm against Indigenous communities, the Visioning B.E.A.R. Circle Intertribal Coalition developed a new method of evaluation. First they acknowledge how their training courses were designed alongside Indigenous and local values. Then they adapted the Principles-Focused Evaluation method to suit their ways of communicating, their power structures and their local knowledge.
Listen to the podcast above to hear Strong Oak explore how coalition came to this process, the reasons for it, and the challenges along the way.
Politics of Translation - or who is heard?
Participatory methods of evaluation are integral to decolonising practice, but it’s also important to ask who is telling the story of the impact of the project or programme - and who is actually being listened to and heard.
Power shapes who is heard and who is not. This is no different in evaluation processes, where the knowledge provided faces repeated practices of translation - first by the design of the process, then by the way in which the responses are understood, then how they are analysed, then how they are publicised or reported, and finally how those outside of the process understand that report. How much is lost in that process of translation?
Towards embodied evaluation processes
As Ruby Quantson Davis argues, one method of decolonising evaluation processes is using an embodied practice. This centre the lived experience as a way of allowing people and participants of projects to determine and shape the evaluation experience.
“A reliance on measuring success and impact against predetermined outcomes creates a linear process of program logic that takes away curiosity and does not allow for emergence of unexpected events, outcomes, or learnings. The cultural practices we work with, particularly in the Global South, are not linear; neither are their ways of learning and knowing. Embodied evaluation of peacebuilding projects must respect and acknowledge this absenceof linearity and integrate into an evaluation process practices that allow for emergence and evolution of local learning. Evaluation in peacebuilding must first and foremost be a learning process that generates insights and stories. These insights can help assess the impact of funded projects.”
Embodied knowledge understands that the way in which we learn and exist in the world is through our own bodies, and the connection those bodies have with the world around us. Embodied knowledge centres interdependence and relationship as way of analysis and interpretation.
In relation to evaluation, embodied evaluation processes are designed around understanding context, needs and power relationships. It allows for a deeper understanding of impact (and what impact actually matters to the people involved), one which gives space for the voices of the oppressed and place through which they can actually be heard and understood.
Indigenous Data Sovereignty and Governance
Indigenous data sovereignty is “indigenous peoples’ right to maintain, control, protect and develop their cultural heritage, traditional knowledge and traditional cultural expressions, as well as their right to maintain, control, protect and develop their intellectual property over these”.
Integral to providing data sovereignty is Indigenous data governance, which can be defined as:
“Data governance is the power and authority over the design, ownership, access to and use of data... For Indigenous peoples, whose traditional modes of governance were disrupted by western modes of democratic governance, re-asserting themselves through self-determined governance structures is critical. Ownership of governance structures commences at the development stage, and continues through the ethics application stage and through the collection, analysis and reporting of data, and through policy translation. Indigenous peoples' ownership is integral to autonomy.”
Within the evaluation context, this is about power over the development of the process, data collection, data analysis and reporting. Instead of evaluation processes being extractive processes where an external body takes information and learning knowledge away from a community to report back to a grant-giving authority, the data is held by the community and they make the decisions on how and what to share.
Indigenous Innovation Initiative
Indigenous Knowledges are plural, and the ways in which evaluation, knowledge and data is collected and understood varies. That’s why the Indigenous Innovation Initiative spent a year slowly developing an evaluation framework - the Inquiry and Learning Bundle that worked for the communities they work with.
Understanding that Indigenous data has often been extracted and and used against the communities they work with, they also developed an Nindokiikayencikewin or the Indigenous Knowledges and Data Governance Protocol.
“Data sovereignty is not just about of First Nation, Inuit and Metis Peoples stewarding their own Knowledges and data - it is about improving their capacity and well-being through meaningful, self-determined and self-governed processes that support their visions and sustainability as communities. Data sovereignty is also about empowering First Nation, Inuit and Metis communities to choose when and how to allow others to reciprocally apply their Knowledges and data in ways that benefit non-Indigenous people - for example, applying Indigenous environmental stewardship practices more broadly.” - Inquiry and Learning Bundle
The image above is a simplification of the the Protocol, which includes guidance on all stages of any evaluation processes, with practical guidance on values and ways of taking part in any evaluation process with the First Nation, Metis and Inuit peoples.
In our next newsletter: Can decolonising evaluation lead to systems change? Learn about how practitioners from around the world are putting into practise decolonising evaluation methods, the challenges that come with it, and the lived experience of doing that work.