Utilizing the Theory of Change as a Powerful Tool for Evaluation

According to Leeuw & Donaldson (2015), various evaluation theories have emerged over the years and have transformed the evaluation discipline in one way or another. Having a defined evaluation approach is instrumental in fully comprehending the intricacies of the evaluation world. One way to understand complex social problems is to use a Theory-based evaluation approach. INTRAC (2017c) states, "A theory-based evaluation is usually based on an explicit Theory of Change (ToC) or Logic Model (LM) that explains the theory of a development intervention or a set  of interventions." As I started honing my evaluation skills, I developed an interest in the utilization of Theory of change and especially when I use the theory-based approach as knowledge base to understand the evaluand. It provides invaluable insights into the evaluands throughout the evaluation cycle. The aim of this blog is, therefore, to raise awareness on the utilization of the ToC as a powerful tool to aid evaluators to unpack the evaluands and build an explanatory account of how a program/project/policy worked and under which assumptions or conditions it succeeded or failed for improvement.

However, the theory-based approach alone cannot comprehensively assess how well a given intervention has worked and in which context. By experience, and as the trailblazers of the Realist Evaluation approach claimed, "Programmes are theories incarnate" (Pawson and Tilley 2004, p.3). Consequently, other evaluation approaches must complement the theory-based approach to build plausible conclusions from evaluation findings. Tilley (2000) claimed that intervention theories are multifaceted and are influenced by many choices that guide their conceptualization and implementation in different contexts. Because theories are never explicit enough to reveal the mechanisms behind the assumptions to examine the relationship among different elements of the program and to test hypotheses about how, for whom, and in which circumstances programs worked or did not work, more work must be done. Delving into the validation of the ToC is critical to measure both the program theory and implementation theory to conclude the conditions and assumptions under which they influence program participants to respond to the intervention and spur complex social changes.

Anderson (2005) posited that a ToC is a suitable tool that helps to create solutions to complex social problems. A ToC articulates how change is expected to happen, appreciates the assumptions for such changes to happen, describes the pathways through which the program results can be achieved in the short-term, mid-term, and long-term, and documents changes as they occur. A ToC also aids in establishing causal linkages and assumptions to understand better the challenges that can hinder the achievement of expected changes and create a shared understanding among all stakeholders regarding program interventions and their objectives. Designing a ToC in a participatory manner forges common understanding, fosters ownership and stimulates motivation among program participants to contribute to long-term impacts (Harries, Hodgson, and Noble 2014). According AECF (2022), "A Theory of Change can refer to the beliefs and assumptions about how a desired change will happen or a goal will be realized." ( AECF 2022, p.2). Over the years, the ToC has emerged as a vital tool in program development forming the basis for strategic planning, monitoring and evaluation of social and environmental programs (Belcher, Bonaiuti and Thiele, 2024). More ideas and innovations continue to emerge, empowering evaluation practitioners to grasp the breadth and depth of complex social programs.

Lessons learned:

 Evaluation stakeholder participants: 

1. The selection of stakeholders is very important: understanding their experience, beliefs, perceptions, attitudes, conviction, opinion about the program is crucial to know how the program worked/ did not work. They are a real fountain of knowledge to the design of a comprehensive ToC.
2. The roles of stakeholders: defining the roles of evaluation stakeholders is pivotal for evaluation planning purposes. It steers the profiling of stakeholders to know better what they contributed to the program, assess what is attributed to the change observed, and make a sound judgment of the program performance. It also clarifies the conditions and assumptions of program interventions to drive social changes as expected.
3. Learning from and reflecting with stakeholders helps to package the evaluation findings of complex social and environemental programs. At the end of the evaluation, it is advisable to validate the evaluation findings with stakeholders for buy-in, ownership and replicability of the interventions if succeeded or refute the assumptions under which the expected change did not happen for redesigning the program.

• I recently evaluated a program that had neither a ToC nor an LM. Still, with the contribution of key stakeholders, mainly the Executive members and different Committee Leaders, I was able to trace out the intervention, appreciate the program's theory, and infer the implementation theory to design a graphic representation of the intervention with explicit assumptions of and factors that influenced how the change was brought about. At the end of the evaluation, the group was able to have its first-ever evaluation report with a ToC and LM. Both the ToC and the LM were validated with/by project beneficiaries during the stakeholder validation workshop. As a matter of fact, the program leadership is in a better position to take remedial action in areas the program poorly performed and redesign the program for improvement.

 Data collection methods and Theory of Change

Focus Group Discussion-Group photo at Mother Africa Need You (MANY) Seed Enterprise Multiplication Farm in Côte d’Ivoire ©TAAT/Atayi Opaluwah.

1. Focus Group Discussion (FGD): This method is very useful in cases where a program lacks a Theory of Change. It forms a solid foundation for designing a ToC. It also contributes to understanding the Why "Program theory" by building and validating the How "implementation theory," as these clarify the ToC and make explicit the LM with a clear description of how the program fits together from inputs, outputs, outcomes, and impact.
2. Document Review (DR): This data collection method is critical in unpacking the list of activities carried out and connecting them to Outputs-Outcomes-Impact to understand how change was brought about. Both the ToC and LM make explicit the logic that supports the program serving as the primary tool for program monitoring and evaluation.

• FGD and DR provide an excellent avenue for collecting qualitative data for further analysis. They also offer opportunities to probe further the planning monitoring and implementation processes of the intervention to validate the theory that guided the intervention. An outcome evaluation of a complex program in Cote d'Ivoire revealed that having a ToC at the program's beginning was instrumental to build an explanatory note of the mechanisms behind the change observed and delivering the necessary resources or inputs to program participants and document the processes by connecting Activities to Outputs then to Outcomes. Hearing the voices from the field against the project planning remains paramount to project success.

References

AECF. (2022). Developin a Theory of Change: Practical guide. [Available online]:aecf-theoryofchange-guidance-2022.pdf

Anderson, A.A. (2005). The Community Builders’ Approach to theory of Change: A practical guide to theory development. The Aspen Institute Roundtable on Community Change. [Available online]: TOCI-final3.indd

Belcher, B.M, Bonaiuti, E., & Thiele, B. (2024). Applying Theory of Change in research program planning: Lessons from CGIAR. Environmental Science and Policy. Vol160. 103850, [Available online]: https://doi.org/10.1016/j.envsci.2024.103850

Harries, E., Hodgson, L. & Noble J. (2014). Creating Your Theory of Change: Introduction to Theory of Change. [Available online]: Center for Theory of Change - Theory of Change Community 

INTRAC for Civil Society. (2017). Different types of Evaluations. [Available online]: https://www.intrac.org/ 

Pawson, R. & Tilley, N. (2004). Realistic Evaluation. London. Sage Publications.

Tilley, N. (2000). Realistic Evaluation: An overview. Nottingham Trent University. [Available online]: https://www.researchgate.net/publication/252160435_Realistic_Evaluation_An_Overview