|Hello <<First Name>>
Greetings from the Stepping Stones team in London! This is the fifth edition of our newsletter. In this issue we look at the process of Monitoring and Evaluation of Stepping Stones projects. By clicking on the underlined words you can access all the relevant documents. The continual process of monitoring and evaluation is essential to the success of every organisation.
This coming Thursday will mark International Women's Day 2012 - which serves to remind us that many M&E tools aren't appropriate to record the subtle changes in power and social relations between men and women. One great resource to help us combat this trend is AWID's M&E wiki. It shows us how different M&E frameworks, approaches, and tools have been adapted by women’s organizations to capture the complexity of changes in women’s rights and gender equality work.
Srilatha Batliwala and Alexandra Pittman list the following objectives that the process of Monitoring and Evaluation should fulfil in "Capturing Change in Women's Realities: A critical overview of current Monitoring and Evaluation Frameworks and Approaches" (2011).
But we all know that achieving all of these objectives is difficult for a number of reasons. In some cases M&E is only an afterthought. It can be badly implemented due to a lack of resources. The pressure to "perform" for donors and funders (in order to gain renewed project funding) can distort the process of evaluation. Below we talk about these challenges and look at ways to overcome them.
We ask Stepping Stones members from around the world to answer questions
about their experiences of Monitoring & Evaluation and ideas for the future
Dr Alice Welbourn,
Director of Salamander Trust & Founder and Director of Stepping Stones
What were the key challenges when monitoring and evaluating Stepping Stones?
We have an on-going challenge with the Stepping Stones programme about how to evaluate it in a way which is straightforward and economic enough for organizations to afford and manage, with limited staffing and budgets; and yet systematic and standardized enough for the results to be considered “robust” by academics and donors. A big challenge! On the one hand the widely and justly acclaimed Randomised Control Trial conducted by the South African Medical Research Council on Stepping Stones (Jewkes et al 2008) was so expensive to run that the programme could only work with two (younger) peer groups, rather than the recommended four peer groups. So it was terrific that this programme achieved the changes that it did. On the other hand, despite the fact that many organizations using Stepping Stones have reported similar findings around the world, their findings have often been dismissed as “anecdotal” because they have not used officially recognized standards for data collection and analysis.
How can we overcome these challenges?
One way forward might be to adopt “combination evaluation”, as we have adopted combination prevention and combination treatment – in recognition that one approach alone is insufficient to cover all contexts. A recent AWID report, based on analysis of 37 European women’s rights organizations receiving funds from the Dutch government, has offered 13 suggestions to organizations working on women’s rights (Srilatha 2011). We think that these suggestions sound very relevant to us in relation to Stepping Stones also. You can find a detailed explanation of these findings and why they are important by clicking here.
Programme Director, Karnataka Health Promotion Trust, India
What tools have you implemented to improve Monitoring & Evaluation?
In our project, we devised a monitoring system for Stepping Stones sessions. The monitoring indicators include tracking attendance of participants, flow of sessions, action plans developed in sessions and changes seen in the community. We have also undertaken an evaluation using quantitative (polling booth surveys) and qualitative (Focus group discussions, most significant story) tools to understand change in knowledge, attitude and practices related to HIV and gender.
I believe that the process adopted in Stepping Stones has the ability to self evaluate the expected outcomes. The first sessions of joys and sorrows of sex or what is love or ideal images or risk taking sessions can be used as baseline. Similar sessions can be done after 6 – 8 months with the same group to track changes in knowledge, attitudes or practices.
What challenges have you faced throughout this implementation?
The key challenge is lack of trained professionals who can make this process more scientific (acceptable to academicians or researchers) and who can analyse the qualitative data generated through these charts and drawings and articulate them in a way that is acceptable to the academic or research world. I think we need to define how this can be done - (what sampling frame to use, which sessions to use, how to analyse the data...? etc) in a way that it can be used by all practitioners. This evaluation can be done by an external researcher or facilitator but he or she can use the data generated by the participants during the Stepping Stones sessions. I think this will also be a cost effective way of doing a evaluation.
"Stepping Stones is the first HIV prevention behavioural intervention to have been subject to the most rigorous level of evaluation in Africa"
(Jewkes et al, Medical Research Centre, South Africa, 2007)
Robyn Drysdale (in photo with Stepping Stones trainer, Henry Oti),
Former Regional HIV/AIDS Prevention Adviser for the Pacific
How was Stepping Stones introduced into the Pacific?
Stepping Stones was introduced in two Pacific island countries (Fiji & Solomon Islands) in 2006 by the Pacific Regional HIV/AIDS project (PRHP) and the Secretariat of the Pacific Community (SPC). As Pacific regional Prevention Adviser with SPC from 2004-2009 I was a member of this team, and became a facilitator and trainer of the program. Based on the initial pilots and the training of facilitators, a Pacific version of the Stepping Stones manual was developed.
How did you develop the M&E toolkit for Stepping Stones in the Pacific Islands?
To fit with the nature of Stepping Stones, all of the M&E methods to be used needed to be ‘participatory’ forms of evaluation. Participatory evaluation attempts to involve as many people with a stake in the project as possible. This includes managers, facilitators, participants and the wider community where Steping Stones has been run. The strengths of participatory evaluation lie in its inclusive approach, where implementers and beneficiaries are given the opportunity to report on project successes or failures. On this basis a range of methods and tools were reviewed, considered and adapted. The resulting toolkit contains 3 simple tools which were selected which are able to collect relevant and useful data on individual and community change. With the tools included in the kit, program implementers are able to learn about how the project is progressing in communities, what changes have resulted from Stepping Stones and ways in which they can improve for the future.
Which M&E tools did you decide to use and why?
1.Community Self Assessment- measuring change and community attitudes. It Allows communities to define their own strengths and weaknesses in relation to HIV and record perceptions of change at the whole community level. This is carried out both before and after the training sessions. It was adapted from the UNAIDS Community Competency Self-Assessment Framework.
2. Gender Scale Assessment (G-scale)– measuring community attitudes towards gender equality (amongst men & women). Simple questionnaire carried out pre and post Stepping Stones (oral or written). All questions relate to gender roles in relationships, the home and community. Adapted from GEM Scale (Project H in South America)
3. The Most Significant Change (MSC) Story Collection Technique - measuring change at the individual and community level.Based on ‘stories’ of significant change coming from the community level. A central part of MSC is an open question to participants, for example: “looking back over the Stepping Stones process, what do you think are some significant changes that have happened to you and your community?” Originally developed in 1996 by Rick Davies.
Example of a post Stepping Stones training interview: “[Before] I felt ashamed to talk about sex and HIV in my village as it is a taboo subject…[.After] I have become much more confident and open and am passionate about spreading the word of HIV. I think it is important to help break down the taboos that will spread HIV”
What were the key challenges of this approach to monitoring and evaluation?
Finding a selection of ‘core’ Stepping Stones groups within countries who will assist to continually motivate and build capacity and support community facilitators. These core group members also need to have the capacity to utilise collected M&E data, which poses challenges.
Maintaining sufficient monitoring and support from regional level – assisting with M&E training and providing support in undertaking community pre and post assessments.
Community Facilitator journal – monitoring tool.
The ‘Community’ facilitator journal format simplifies the collection of information as you conduct Stepping Stones workshops. (This journal can be downloaded free from the Stepping Stones Feedback website!)
Monitoring and evaluation toolkit for community based Stepping Stones programs in the Pacific - Practical Guide
Monitoring & Evaluation tools to help you
This Monitoring and Evaluation (M&E) Toolkit is a practical guide on how to gather information on individual and community change that has happened as a result of the Stepping Stones project. This can be downloaded by clicking here.
(Note: In the Pacific Stepping Stones is now administered by the Foundation for Peoples of the South Pacific International (FSPI) with support from SPC.)
USING OUTCOME MAPPING WITH STEPPING STONES IN CENTRAL AMERICA
Between 2009 – 2012 a team of consultants made up of Javier Pacheco Arriera, Rosa Isabel Garza, Amandine Bollinger, Marjolein Jacobs, Carlina Velez and an excellent team of Stepping Stones facilitators and promoters, worked to faciliatate a process of adapting Outcome Mapping to create a set monitoring and evaluation tools suitable for Stepping Stones in Central America. Outcome Mapping was considered to be a particuarly suitable tool to use with Stepping Stones because of its ability to record behavioural changes at various stages. This work has resulted in the development of a systematic process of monitoring and evaluating Stepping Stones programs at a regional level.
“Designing the project reflected not only a labour of love but also a consolidation of many different, vital aspects. A key focus for our work was clarity;clarity in terms of the relationships and agreements with our funders, clarity as an essential requirement for our technical and administrative team, and clarity in terms of that which we can offer communities who are just beginning the Stepping Stones programme.” Javier Pacheco Arriera, Outcome Mapping Facilitator
To read a report written by Amandine Bollinger about this process, click here
First Outcome Mapping Workshop held in El Salvador in 2009
A RECENT STEPPING STONES EVALUATION
Last year, Bradley et al. conducted an evaluation of Stepping Stones in India for BioMedCentral (BMC).
This evaluation was created to measure three main aims:
To measure the impact of Stepping Stones training on individual participants
To measure how Stepping Stones impacted the close friends and contacts of these trainees
To measure the impact that Stepping Stones training has made a community level
1.In depth interviews with past Stepping Stones trainees and their close friends
2.Confidential polling booth surveys with past trainees, general population members from their communities as well as from communities with no such training.
Click to read the evaluation in full: "Evaluation of Stepping Stones as a tool for changing knowledge, attitudes and behaviours associated with gender, relationships and HIV risk in Karnataka, India."
The training made a clear and lasting impact on the individuals that undertook it although it did not always tranlate into community level actions
Stepping Stones training changed many people's behaviour with regard to HIV and gender issues
Stepping went on to be implemeted at an national level in India.
One interesting finding from this evaluation was that, in some cases, the Stepping Stones workshops in Karnataka appeared to
significantly change participants behaviour whilst participants attitudes remained unchanged.
This goes against the classic public health model which has assumed a linear progression from:
Intervention >> attitude change >> behaviour change
In the case of HIV intervention this model tends to assume the following:
In our experience, human responses to intervention can be very complex and the linear, causal model of change (shown above) cannot fully understand other important factors. For example, many people know smoking is bad for their health - they are told by their doctors why they should stop smoking - and yet they continue to smoke... Why?
What changes do you think come first - changes in behaviour or changes in attitude? Can change be multi-directional? Stepping Stones Feedback is interested in your experiences & opinions about the key challenges you have faced. Please tell us by writing in the Stepping Stones online forum.If you are not already a member please join by clicking here and creating a profile. We really welcome your ideas and suggestions for moving forward with this work. Do please write in and we will share your comments throughout the network. We look forward to hearing from you!
Recommended M&E Resources and Other useful websites