KIT provides a full range of evaluations, from informal participatory reviews, to large-scale, rigorous impact evaluations. We help our partners and clients to understand whether and how their programmes, interventions or processes work – and we assess their impact.
Tailor-made evaluation models
In a climate of diminishing funding, health programmes are increasingly being asked to demonstrate their effectiveness. Evaluation is a powerful tool to assess the social and economic impacts, and to learn what works and why. Our expertise and track record in the area of evaluation make us well-equipped to evaluate your work.
KIT uses mixed methods in design, data collection and analysis. We pay special attention to using the most suitable models, frameworks and tools and to assist our clients in translating findings into action.
Characteristics of the KIT approach:
- We involve key stakeholders in the different evaluation phases including design, data collection and analysis, translation, validation, contextualization and publishing.
- We bring together qualitative and quantitative researchers to allow for genuine integration of methods from the very start. This approach facilitates broader thinking about evaluation and genuine collaboration between experts with complementary expertise.
- We work hand-in-hand with local researchers and/or evaluators in facilitating capacity building.
In depth: The KIT approach to impact evaluation? It’s all about mixed methods.
Evaluating the impact of health programmes informs effective policy-making and evidence-based programme design. The mixed methods approach of impact evaluation means combining breadth and depth of measuring, to achieve new insight into how well health programmes work. Two general groups of methods can be thought of in the mixed approach: “quantitative approaches” that determine whether there was an effect – for example by comparing the situation before/after and with/without the intervention; and “qualitative approaches,” which are more appropriate to explore how, why, for whom the intervention worked. Combining both approaches in a mixed-methods evaluation supports a comprehensive understanding of the extent and reasons for change in a health programme.
At KIT we have a vast experience in different types of mixed-methods evaluation. This includes impact evaluations, as well as broader ‘programme’ evaluations. Broader evaluations include the review of results against objectives – effectiveness – while also looking into aspects of relevance, sustainability, and efficiency in the given health context. Evaluations typically serve two main functions: accountability and learning. While the primary focus of the evaluations that we conduct may often be on accountability towards donors, we also strive to ensure that our evaluations function as learning opportunities for policy-making and programme design. As such, we also hope to ultimately contribute towards accountability to those most directly affected by programmes – the primary beneficiaries (be it community members or implementing staff).
There are various methodologies that can be used to conduct an impact evaluation. “Counter-factual” methodologies, which mimic the type of trials conducted to assess the benefits of new medicines in clinical research, are regarded by many as the most robust for impact evaluations. At KIT we value their robustness and have the necessary expertise in house to conduct them. However, the fact also holds true that real-life program implementation may not always be conducive for these type of methodologies, and that simpler designs can be required. Ultimately, evaluation design is dependent on context, but we believe in the added value of including qualitative and theory-based methodologies as part of all impact evaluations. When qualitative and quantitative approaches are combined, it is important to mix methods from the start of the evaluation – and at each step of the study – to ensure that the right questions are asked to explain results. The guide below – also outlined visually above – shows our approach, step-by-step.
KIT’s Impact Evaluations: a step-by-step guide
Step 1: Define evaluation needs
This is the time to fully understand the aim and scope of the evaluation. This starts with understanding the change that a programme aims to achieve, and the pathways to get there. We facilitate a process where the stakeholders’ expectations are clarified and consensus is built around the core aim and scope of the evaluation.
Step 2: Identify and review existing sources of data
Based on the core aims and scope identified, we proceed to define the evaluation: What are the available resources for the evaluation? What stage in its life-cycle is the project currently in? What data has been generated by the project and can these be meaningfully used for the evaluation? At this stage we also make an inventory of key sources of external data such as health records (HMIS data), population based surveys (DHS/MICS), and academic literature. On a case-by-case basis we appraise data reliability and assess the possible contribution of each data source to our information needs.
- Step 2: Mixing moment: During this stage it is important to take stock of all qualitative and quantitative data available and to start identifying data gaps to address the evaluation needs.
Step 3: Exploratory needs assessment
In some cases field visits with stakeholders and the study population may be needed to understand the reality on the ground. During this step we consult a wider range of stakeholders to try to understand the health programme from their perspective. Ideally this step should also clarify what people on the ground think about the changes from the health programme – including the pathways to arrive at that change. This step can also be a way to get buy-in and support for evaluation activities and ultimately improve the results of the evaluation.
For theory-based evaluations this can be an ideal time to identify the theories underlying the health programme. This may start with interviews with stakeholders to get their perspectives on what causes change, and link these to relevant existing theories.
During an exploratory needs assessment we may need to go back to the data sources identified earlier (Step 2) and re-assess them in the light of new leads identified on the grounds. Depending on the complexity of the intervention and evaluation, this may need a few rounds of iteration.
Step 4: Finalise evaluation question
All preparatory work and consultations then enable the finalisation of the evaluation objectives, questions, themes and variables. We strive for participatory consensus-building with all relevant stakeholders.
- Step 4: Mixing moment: The final evaluation question should be able to build a holistic understanding of the health programme. There is a risk that various (sub)questions correspond to a given methodology (quantitative or qualitative) causing parallel streams of research that “do not speak to each other.” We try to avoid this by reflecting on the qualitative and quantitative information needs for each (sub)question.
Step 5: Study design and tools
While randomised control trials are regarded by many as the “gold standard” for impact evaluations, there are a number of other “counter-factual methodologies” which can be considered, such as propensity score matching, regression discontinuity, and difference-in-difference. However the complexity of public health programmes may limit their feasibility and suitability, as they do not lead to understanding what causes change in a health system. Theory based impact evaluations (such as process tracing, congruence analysis, qualitative comparative analysis) are best suited for this purpose. They involve the articulation and testing of causal models for change to gauge the contribution of a programme towards better health outcomes, also taking other external factors into account. However theory-based methodologies are very intensive and time-consuming. Given the limitations of both counter-factual and theory-based methodologies, simpler designs (before and after, with and without controls) are sometimes the only available option in practice.
Regardless of the study design, mixing qualitative and quantitative methods will enable a deeper and more contextualised understanding of the impact reached. Research tools (e.g. survey questionnaires or qualitative data collection guides) should be as valid and reliable as possible, and field procedures should be clearly described. For quantitative data collection, tools should preferably build on existing validated tools. For qualitative data collection, guides usually have to be designed from scratch, in order to be fully relevant in the given context.
- Mixing moment: Tools for quantitative and qualitative data collection should be complimentary and ensure that all evaluation questions can be answered. The order of qualitative and quantitative data collection may vary and can be done in different combinations. These need to be defined ahead of data collection.
Step 6: Ethical review
Depending on scope and type of data collection needed for an evaluation, ethical review may or may not be required for an evaluation. According to the latest international ethical guidelines, double review should be sought – both in the host country as well as in the sponsoring country. The KIT research ethics committee was especially set up to review public health research and includes a multi-disciplinary team of reviewers, who review protocols both from a technical and ethical perspective. Local ethical review, wherever possible, plays a crucial role in ensuring the contextualisation of the design and tools used for an evaluation.
Step 7: Pilot test
The first draft of a tool should be tested in a small number of respondents after which adaptations will follow. Pilot testing on the target population is intended to test the comprehensibility, relevance, acceptability, and feasibility of the evaluation instruments and terms used. Conducting a first pilot with peers can be very useful and reveal a number of issues. In a pilot, after participants have answered all questions they should be asked about their experience in as much detail as necessary to enable changes. Pilot testing is crucial to ensure that tools are adequately adapted to a local context.
Step 8: Field data collection
When an instrument is considered to be satisfactory after one or more rounds of pilot testing, it is then applied to the target population. Field staff are selected according to competence criteria, with clear roles and responsibilities, and must be adequately trained.
Depending on the design, data collection can be a repeated process: if the evaluation design is done over time (such as before/after comparisons) then data collection is conducted at multiple time-points. When a baseline study has been carried out, experiences with data collection as well as results from baseline analyses may inform data collection. Programme evaluations as well as theory-based evaluations are typically conducted at one time-period only.
- Mixing moment: Qualitative data collection (focus group discussions, in-depth interviews) could take place at start of a study to explore a given issue, and then based on the qualitative results, quantitative data collection may be adapted to measure the effect size and distribution of those issues. The reverse can also be done: first the quantitative data collection, then qualitative to explain the quantitative results.
Step 9: Analysis
Mixed-methods analysis is typically conducted by repeated analysis. Qualitative data analysis starts at the time of field work to inform on-going data collection and ensure that all emerging themes are being explored. This is followed by further rounds with a preliminary description of both quantitative and transcribed qualitative data. The following step includes statistical (descriptive and inferential) analysis of quantitative data and synthesis of the qualitative data (key themes).
We then proceed with a first stage of mixing: putting together quantitative and qualitative data on the same topic to answer each of the evaluation questions comprehensively. This helps to define further higher level analyses to fulfill the aim of the evaluation. Further analyses looking at issues such as how, why, and for whom the intervention worked can also help understand causality, linking the health programme’s intervention efforts with outcomes.
Participatory data analysis workshops with key stakeholders involved in an intervention can also be used to provide a powerful opportunity to contextualise and validate research findings, and to translate these findings into actionable recommendations.
- Mixing moment – The analysis phase is the most important mixing moment, ultimately leading to evaluation results which comprehensively address evaluation needs, combining the generalisability of quantitative methods with the contextualisation of qualitative methods.
Step 10: Participatory product development and dissemination
For effective dissemination, evaluation findings need to be communicated in forms that enable potential ‘users’ to find, understand, and use them. This can be challenging, especially for impact evaluation where evaluators generally interact with numerous different target groups. Participatory approaches are used to translate evaluation findings into different formats appropriate to the respective target audience.