The ToR for the evaluation of an outcome mapping project in Cambodia stipulated “the successful candidate will work with four staff members and four community members”. This was irresistibly intriguing. I applied.

The NGO and I chose Outcome Harvesting as the most appropriate methodology. But how to use my team of 8? 8 people with Khmer as first language and who understand Cambodian village life in a way I never can as co-harvesters was a huge positive and the sheer volume of stories would be massive. However none of them spoke English, none had evaluation experience and their understanding of outcomes was not deeply nuanced. Four were community members without tertiary education, no training in outcomes or outcome harvesting, even how to do an open ended interview. Also while 8 extra interviewers would increase my data the sheer volume was going to be hard to manage. Time wise 8 non English speakers, most of whom were stronger with spoken than written language would make it hard to capture and process the outcomes they collected. Hmmm.

The benefits hugely outweighed the costs. It was really enjoyable and we finished with a high quality, very useful harvest. Here’s how:

Pre-Evaluation:

• As is usual with an Outcome Harvest I spent a lot of time on Skype with the project manager, refining evaluation questions, working through outcome statements (done with his team), and setting up the timetable.
• I got a copy the qualitative research software NVIvo and learned how to use it. It did not take too long and was an excellent investment of time.
• I arranged a translator totally comfortable in Khmer, English and computers.

On arrival.

The two programmed days of team training were a good test in creatively and simply transmitting difficult ideas ( and remember everything was translated). I used games, role plays, video clips, stories, a little power point and more games. Benefits were :

• Over two days training, having fun and sharing delicious Cambodian food together we started becoming a team.
• We developed a shared understanding of key concepts, a common language
• We represented, refined and understood better the key evaluation questions.
• I began to understand the strengths and weaknesses of each person.
• I presented NVivo and the process we’d use and we made a plan together on how to capture our outcome stories. It felt very participatory.

Content-wise we achieved a clear understanding of what outcomes are, what significance and contribution mean, how to probe for them without using closed questions, lots of practice, facilitating conversation, allowing outcomes to emerge and practice in ways to re-tell and capture outcomes.

Field time.

Consisted of our team travelling to three project sites and with the manager at each site deciding on villages to visit. Each morning we made a plan of who would go where and who they might talk to. I made sure to leave unscheduled spaces to let them allow unplanned, useful conversations to unfold. Initially some of my team were not confident to go alone so visited went in pairs but after a few days everyone just headed to villages by themselves. Mid -afternoon we reconvened and re-told our stories. My (amazing) translator could listen in Khmer and type in English. I sat next to her and simply read as she typed stories straight into the NVivo database which saved time and allowed narrative to flow than her having to stop the speakers to tell me in English.

All ten of us then probed the stories together asking what the significance was, what the project contribution was what else might have been asked, who else should be talked to based on the outcomes in this story. The whole process of listening to and probing stories took about two hours a day, but it was well spent time because we gained a better understanding of our community and context and what outcome based development is and we ended with stories in the NVivo database.

Post field:

Back in Phnom Penh I had one day with the team. Central to NVIvo is the node- structure- the groups into which outcomes fall. I deliberately avoided pre-setting my node structure, preferring to allow a 'conversation with the context' rather than set up an interrogation.

I showed the team how NVivo works- using metaphors like harvesting as finding and plucking tropical fruit and the nodes as baskets of similar fruit. I then set my team the task of setting up the node structure, figuring that they might understand deep meaning of all our combined stories in ways I did not. So the way we grouped (and understood) outcomes emerged un-planned from the team and the context

Using their node structure (modified slightly) I spent a busy weekend coding- going through all their stories, finding the relevant text in which outcomes emerged and dragging and dropping into the right database ‘basket’. This was a lot of work but gave me a really good, in depth knowledge of our entire database ready for a big presentation to multiple stakeholders on the Monday. And coding all the stories myself meant I began to appreciate patterns in ways I otherwise would not have. various ways NVivo can compare, store, query group and display data were very useful. Standard and novel answers to evaluation questions emerged, but I felt this happened through me, not via the software. I asked the questions,  NVivo was the office boy doing the legwork

The big take homes:

• A local team can hugely enhance an Outcome Harvest. For me the less accessible a language and culture, the more useful a local team is. The volume of stories is always a plus. Formal education is not essential. Good relationships with them is. And most developing country locals are strong on the essential harvesting skills of narrative and relationships.
• Programme time before hand to all get on the same page before going to the field and time in the field to unravel significance and contribution
• A team will generate a huge amount of data, probably more than your brain alone can manage. Volume and pattern is ideal for computers. Experiment with qualitative research software is ideal to manage and process data.
• Make sure the human understanding of your data is done by humans. Use the team and your own brain. Do not outsource this to the software.

want to talk about it?

I loved doing this Harvest and am happy to talk abut it to anyone. Please feel free to contact me to discuss any aspect of this- jephmathias@gmail.com. You'll find various dimensions written up on my website www.unpredictable.co Try http://www.unpredictable.co/news/2016/9/23/my-evaluation-metaphor-massage for a start.