Daniel is a reluctant evaluator, his passion is monitoring with a focus on listening to and learning from those who deliver the support and those who matter most- their clients or ultimate beneficiaries. He strongly believes it is to them, not donors, organisations and programmes should be primarily accountable.
Daniel’s skills are in advising and mentoring in:
• facilitating development and reviews of theories of change and results frameworks, preferably one or the other, certainly not both;
• reviewing, establishing and developing thoughtful monitoring and learning processes and products in organisations and programmes;
• fostering cross team and portfolio learning; and
• designing, oversighting and leading different types of evaluations – ex ante, process and impact - that are utilisation focussed.
Daniel holds an MSc in Agricultural Economics, with a focus on Agrarian Development Overseas, from London University as well as a BA in Geography from the School of African and Asian Studies, University of Sussex.
He lives North of London with his Mosotho wife, Tsepe and has two children – Thabo and Ella. He plays tennis and volunteers at the King’s College African Leadership Centre, University of London and the Woodland Trust.
Posté le 19/04/2024
Dear Rahel,
Thanks for posting this blog about how far "we" have come on impact evaluation. Let me be terse with my answer: not much, if at all. And for the following three reasons:
- CGD's "When Will We Ever Learn" (WWWEL) is a throw back to Vedungs' first scientific wave of evaluation - Vedung, E. (2010) Four Waves of Evaluation Diffusion, Evaluation, Sage Publications, 16: 263 pp. 263-277. During the 1960s and even earlier, advanced evaluative thinking and practice was driven by a notion of scientification of public policy and public administration. It was argued this would make government more rational, scientific and grounded in facts. Its technocratic thrust sought to isolate public policy decisions from the messy, complex world we live in. Evaluation was to be performed by professional academic researchers (often masquerading as evaluators).Spitting roast for the labs and units you list, and many others. Towards the mid-1970s, confidence in experimental evaluation faded however. Voices started communicating how Evaluation should be more diverse and inclusive. Those other than academic researchers should be involved. Ring bells for today's debates on de-colonisation, localisation and Indigenous Evaluation?
2. CGD's self-serving basic thesis:
- "persistent shortcomings in our knowledge of the effects of social policies and programs reflect a gap in both the quantity and quality of impact evaluations.’
- the authors argued: An “evaluation gap” has emerged because governments, official donors, and other funders do not demand or produce enough impact evaluations and because those that are conducted are often methodologically flawed.” They ascribe the evaluation gap to the public good nature of impact measurement; and
- "that governments and development agencies are better at monitoring and process evaluations than at accountability or measuring impact"’ - this may be so but, monitoring, long neglected by the evaluation community, as practiced by most govts and dev agencies, is done far from well and is deliberately held down as routine reporting process (pers comm Michael Quinn Patton, April 2024).
James Morton in his 2009 paper "Why We Will Never Learn" provides a wonderfully lettered critique of the above: the Public Good concept is a favourite resort of academics making the case for public funding of their research. It has the politically useful characteristic of avoiding blame. No one is at fault for the ‘evaluation gap’ if evaluation is, by very its nature, something that will be underfunded. Comfortable as this is, there are immediate problems. For example, it is difficult to argue that accountability is a public good. Why does the funding agency concerned not have a direct, private-good interest in accountability?
Having effectively sidelined Monitoring and Processes, WWWEL goes on to focus, almost entirely, on measuring outcomes and impact. This left the "monitoring gap" conveniently alone. While avoiding any discussion of methodologies: randomised control trials, quasi experimental double-difference, etc. many discussions WWWEL encouraged were the abstruse, even semantic nature of the technical debates which dominate discussion about impact measurement.
3. Pawson and Tilley's expose - through their masterful 1997 publication "Realistic Evaluation" of experimentalists and RCT's intrinsic limits as defined by its narrow use based on the deficiency of its external validity. They challenge orthodox view of experimentation: the construction of equivalent experimental and control groups, the application of interventions to the experimental group only and comparisons of the changes that have taken place in the experimental and control groups as a method of finding out what effect the intervention has had. Their position throws into doubt experimental methods of finding out which programmes do and which do not produce intended and unintended consequences. They maintain it not to be a sound way of deriving sensible lessons for policy and practice.
In sum then, CGD's proposition of RCTs, to cite Paul Krugman. is like a cockroach policy: it was flushed away in the 1970's but returned forty years later along with its significant limits intact; and CGD missed the most significant gap. From the above, one could get the impression that development aid has lost the capacity to learn: it suppresses, not takes heed of, lessons.
I hope the above is seen as a constructive contribution to the debate your blog provokes; and my seeming pessimism simply qualifies my optimism - a book was launched yesterday on monitoring systems in Africa.
Best wishes and good luck,
Daniel
United Kingdom
Daniel Ticehurst
Monitoring > Evaluation Specialist
freelance
Posté le 23/08/2024
J'interviens simplement, comme Silva, pour féliciter Musta d'avoir soulevé un point aussi important. La valeur marginale apparente et les coûts d'opportunité élevés des évaluations environnementales.
Lors de la conférence de la Société européenne d'évaluation de 2022, le discours d'Estelle Raimondo et de Peter Dahler-Larsen a été frappant. Ils ont présenté une analyse intéressante sur l'application indiscriminée et les rendements décroissants de la pratique du retard par son utilisation « performative ». La capture bureaucratique.
Certains affirment que les évaluations de l'évaluabilité sont le moindre des soucis de la communauté de l'évaluation d'aujourd'hui.
La référence à la façon dont « ....parfois, les agences peuvent réduire le risque de réputation et tirer leur légitimité de l'existence d'un système d'évaluation plutôt que de son utilisation » rappelle l'analogie faite en 1903 par le célèbre classiciste et poète AE Housman :
« ...les gentlemen qui utilisent des manuscrits comme les alcooliques utilisent des lampadaires, non pas pour les éclairer sur leur chemin mais pour dissimuler leur instabilité ».
United Kingdom
Daniel Ticehurst
Monitoring > Evaluation Specialist
freelance
Posté le 06/08/2024
Chère Amy
Merci d'avoir pris le temps de lire et de répondre.
Toutes mes excuses, mais permettez-moi d'être franche et honnête.....
Merci beaucoup d'avoir expliqué ce que M. Scriven a écrit. Je comprends maintenant. Cela dit, je n'ai toujours pas compris l'importance et la signification de ce qu'il a écrit. La maternité et la tarte aux pommes, c'est intelligent mais c'est un peu mince 😉
En ce qui concerne les évaluations de l'évaluabilité elles-mêmes, et comme j'y ai fait allusion, l'objectif et la portée de l'enquête d'une EE semblent faire partie intégrante de ce que la plupart des gens appelleraient une évaluation ex ante compétente. Comme vous le dites, il est bon d'avoir un ou deux évaluateurs dans l'équipe, mais... Comment ne pas examiner l'"évaluabilité" de l'investissement en l'évaluant et en examinant le raisonnement probant qui sous-tend sa raison d'être et ses facteurs de conception, y compris un cahier des charges et/ou un cadre de résultats ? Ou bien sommes-nous en train de dire qu'une évaluation qui recommande qu'un investissement ne soit pas digne d'intérêt peut être jugée évaluable ou, en fait, vice versa (en supposant que l'EE soit réalisée après l'évaluation) ?
Ainsi, et comme le souligne Hadera, la valeur ajoutée générée par la réalisation d'une EE discrète - certains diraient artificielle - uniquement par des évaluateurs semble au mieux marginale, fragmente potentiellement l'équipe et apparaît comme plutôt extravagante et indulgente.
Merci beaucoup pour cet article et pour les discussions qui ont suscité des interrogations, des débats, du scepticisme et des doutes sur les évaluations de l'évaluabilité.
Avec mes meilleurs vœux,
Daniel
United Kingdom
Daniel Ticehurst
Monitoring > Evaluation Specialist
freelance
Posté le 31/07/2024
Chère Amy,
Merci d'avoir posté cette discussion. Je me souviens très bien avoir lu la synthèse de la littérature de Rick D en 2013. Voici les quatre observations que j'ai formulées :
Enfin, un peu d'aide : Vous avez cité Michael Scriven qui a dit "L'évaluabilité est analogue à l'obligation d'entretien d'une nouvelle voiture et peut être considérée comme "le premier commandement en matière de redevabilité". Je sais qu'il s'agit d'une phrase importante, mais je ne la comprends pas et je ne saisis pas son importance. Que pensez-vous qu'il veuille dire ?
Meilleurs vœux et merci encore,
Daniel