Impact Evaluation of UNICEF Nigeria WASH program

News

26 February 2014

From January till May a multi-disciplinary team of KIT is working in Nigeria, to carry out an impact evaluate of a large UNICEF country program on Water, Sanitation and Hygiene (WASH). The program is implemented in all 36 states of Nigeria and ended after 4 years on December 31st 2013. The findings of the impact evaluation will inform national policy and the the implementation of the next UNICEF country program 2014-2018. Evidence of the benefits of the WASH will be valuable for advocacy efforts directed at the federal and state governments.

Impact and effectiveness: what worked and what did not?

Madeleen Wegelin from KIT: ‘Access to clean Water, Sanitation and Hygiene (WASH) saves lives. We are excited that UNICEF selected us for this extensive evaluation. We are keen to contribute to development and implementation of effective future WASH policies in Nigeria – by providing evidence on if it worked, and if so: why, for whom, where.’

The WASH program of UNICEF was carried out at local community level, schools and health centers in close cooperation with local community members, State and Local Government support. The evaluation serves two main purposes:

  • To determine the effectiveness and outcomes of the WASH intervention and to identify what worked well and mechanisms that made it work, in order to learn and improve effectiveness for scaling up.
  • To assess the effects of the WASH interventions supported by UNICEF in order to identify opportunities to improve impact.

KIT’s approach: Stakeholder Participation is Key

KIT considers the involvement and participation of stakeholders, including the target group as important in enhancing the transparency, validity, reliability and usability of the evaluation results and essential to its success. As such it is important to include caretakers in the household – usually the mothers – those in the community who will motivate the households to participate, the artisans who will need to maintain WASH structures later, and the implementing agencies at Local Government level.

We asked the responsible in UNICEF why they selected KIT for this assignment: “you knew how to crack the nut”. Meaning that such an impact evaluation is complex, as (changing) behavior around water and sanitation is complex and because evaluating “impact” is more than measuring if more boreholes were built. It needs information on what worked why for whom – if it worked. Also, there are many methodological challenges if there are no baseline data, and in finding good “counterfactuals”.

Scale and design of the evaluation

The Impact Evaluation is carried out in 6 states in a quasi-experimental design as no baseline data were available. This is done by employing a so-called mixed methods approach – a quantitative household survey combined with qualitative data collection through In-Depth Interviews and focus group discussion. We will combine “hard” data with insight from real field experience of those people involved in implementing this UNICEF program and the ultimate beneficiaries.

We will interview the ultimate beneficiaries in the householdsthrough structured questionnaire in a sample of 1100 households; to quantify if they did benefit, and if there were indeed effects.

We will also interview, through focus group discussions and in-depth interviews:

  • policy makers in the Ministries of Health and Water Resources, State Rural Water Supply and Sanitation Agency and Local Government Area WASH teams to asses if the UNICEF approach was appropriate;
  • facilitators of Community Led Total Sanitation if they were supported adequately by UNICEF;
  • key informants in the communities and Community Based Organizations to appreciate their perception of the effects in WASH – but also in terms of behavior and community organization, and barriers to sustainability;
  • health center workers and possibly head teachers in order to gain more insight into the progress made by the WASH programme.

The Team:

KIT has a highly qualified, multidisciplinary team working on impact evaluations, containing epidemiologists as well as statisticians, economists, and social scientists, all with a vast experience in impact evaluations in different fields of work. Both in quantitative as well as in qualitative impact evaluation methodologies. For this evaluation we have put together a team of experienced experts with highly complementary skill sets. Our experts:

  • Jurriën Toonen: Team leader with extensive monitoring and evaluation experience, who deals with the design issues;
  • Ngozi Akwataghibe: A well-rounded public health expert who has led several projects in Nigeria. She will coordinate the in-country fieldwork;
  • Madeleen Wegelin: A WASH expert who will serve as a reference person on WASH-related issues relevant to the evaluation, and who leads the qualitative component;
  • Liezel Wolmarans: A statistician with extensive survey experience, who leads the quantitative component;
  • Kelsey Vaughan: A health economist who will lead the cost-effectiveness analysis;
  • A quality assurance team: Fred Zaal: An experienced evaluator for the qualitative component, and Sandra Alba: An epidemiologist for the quantitative component.

Contact
Would you like to know more about this project or KIT’s expertise on Impact Evaluation and WASH? Please contact: Jurriën Toonen, j.toonen@kit.nl