‘These laws of information are among the many paradoxical theories of resistentialism which posit that things are against us‘.

This article by KIT Royal Tropical Institute and ITAD published in the BMJ Global Health journal provides useful insights on how these laws of information can help generate useful information for decision-makers while conducting complex evaluations

It highlights the difficulties funders, implementors and evaluators face when evaluating a programme, and it provides important recommendations on how to effectively evaluate complex interventions, based on Finagle’s laws of information.

Drawing on their experience

In 2019, Itad and KIT were commissioned by the Bill & Melinda Gates Foundation to evaluate GRID3‘s, ‘Geo-Referenced Infrastructure and Demographic Data for Development’, use and impact between 2012 and 2019. The programme was implemented in Nigeria’s northern states to support polio and measles immunisation campaigns.

These recommendations are based on the difficulties the evaluators faced while evaluating this innovative programme providing demographic and geographical information to support the delivery of health services.

According to co-author Callum Taylor of Itad, “The GRID3 programme presented a series of interesting challenges to us as evaluators. We are pleased to have turned those challenges into lessons for supporting evidence-informed decision-making through evaluations of complex systems.”

The authors recommend that: 

1. Funders engage evaluators before the start of an evaluation to enable evaluators to influence study design and data collection; 
2. Evaluators interact with funders to understand their information needs, manage expectations regarding the scope of evaluations, and disentangle between impact and process questions; 
3. Implementers and funders use theories of change to make the intervention’s assumptions explicit and to develop monitoring systems that inform on operational processes (not only fiduciary risk); and 
4. Funders and evaluators work together to develop evaluation plans that incorporate a spectrum of studies from smaller-scale experimental studies to larger-scale observational studies. 

“Evaluating elaborate interventions is a complex endeavour by definition. By reflecting candidly on our difficulties, we hope we can help funders, evaluators, and implementers to work together to maximise the usefulness of their evaluations,” says Sandra Alba, co-author and senior advisor at KIT.

You can find a summary by ITAD here.