John O’Connor, a consultant at Ceres Management in the UK, recently offered his professional skills  GoPhilanthropic’s partners TARA Shelter in New Delhi.  John’s practice focuses on improving how we measure the impact of people development initiatives.  As is the case with most small non-profits, TARA is strapped for both resources.  Even the thought of formally measuring or evaluating their success is a drain.

John writes about his experience at TARA applying a very creative and effective measurement tool called Measuring Significant Change.  The process involves involves collecting significant stories from those involved a program–in this case, from the boys residing at the shelter.  Through a process of selecting which ones reveal the most significant change, the impact of a program begins to reveal itself against the rich backdrop of complex variables at play.

September 2012,

On a recent Air India flight to London I watched the movie ‘The Best Exotic Marigold Hotel.’  There wasn’t much else and in fact I had snubbed it on the outward flight. To my surprise I really enjoyed it. Why? It was a profusion of high energy, colour, assorted cultures, and eccentric characters all rolled together in a richly exotic Indian setting. And best of all, the ending wasn’t made in Hollywood.

But what has this got to do with evaluation? Well the answer lies in the energy, enthusiasm, and teamwork that I unexpectedly found in a recent evaluation project that as it happens, also took place in India.  It’s not often we get to do what we really love to, much less in an exotic location and with persons who are just as passionate about evaluation as you are.

How it all started

GoPhilanthropic is a US-­‐based Foundation that connects donors to carefully selected grassroots organisations in ‘developing’ settings. Tara Homes is one of these organisations, a tiny NGO that provides shelter for vulnerable children in New Delhi, India.

Tara is both pragmatic and forward-­‐looking in how it works. Not to be satisfied with the inspiring results it already delivers, Tara wants to more fully understand the impact of its projects and in addition to be truly accountable to its donors and sponsors.  Despite the logical worries of the time and energy it might take, Tara not only bought into evaluation but also the need to ‘up-­‐ skill’ its staff in ways to conduct evaluation.  

Ceres Management through GoPhilanthropic was offered the opportunity to help Tara achieve its evaluation goals. And like the success of the ‘Marigold Hotel, ’ Tara embodied the same critical ingredients: energetic, full of hope, unafraid, collaborative, and willing to ‘have a go’.

Why Most Significant Change (MSC)?

Evaluating in unconventional settings…

Evaluating the impact of conventional training & development programmes is usually fairly straightforward. We are trained to begin with a set of pre-­‐determined objectives and success indicators that makes evaluating reasonable straightforward.  Programs like TARA however are not conventional. They operate within loose and sometimes chaotic boundaries where unintentional, complex and diverse individual outcomes are the norm.  Obviously, this poses a huge challenge to traditional evaluation techniques.

At the core of the Most Significant Change (MSC) technique is the selection of the most significant changes experienced by those impacted by Tara’s programmes.

To measure unpredictable and potentially hidden impact we selected the Most Significant Change (MSC) technique. Although MSC is a radical departure from conventional evaluation methods it was chosen for Tara specifically because it is story-­‐based, is easy to understand, requires no special skills, and can deliver a rich picture of what’s really happening.

So what happened?

Over 5 days, including 2 workshops and 1-1 coaching, we covered the benefits of evaluating, how MSC works, we conducted a role-play and discussed potential risks.
conducting the 2-day MSC workshop
The Tara staff  decided to pilot MSC for a 3-month period, collecting and selecting most significant change stories in that timeframe. During the pilot we will monitor progress, issues and challenges via regular virtual coaching and support sessions. At the end of the pilot we will review the entire experience, consider lessons learned and make amendments for future MSC use.

Stay tuned for the results!