I have been involved in evaluation of research quality for the past 12 year (research papers as an editor also of projects and programs
Posted on 26/05/2023
Thanks, Seda for your important question. As the Guidelines state several times, they were informed by the International Development Research Centre (IDRC) RQ+ Assessment Instrument (www.idrc.ca/RQplus). Hence some useful ideas and suggestions from a development organization are an integral part of the Guidelines.
Perhaps the easiest way to answer your question is to use Table 7 on Pg. 19 Qualitative data themes, indicators per Quality of Science dimension with assessment criteria. This Table was developed for evaluating CGIAR research for development projects. As far as I can see, most of the themes and indicators of quality in a science-based research for development project are just as relevant to evaluating quality in a development project. Under design, as an evaluator I would want to know whether the design was coherent and clear and the methodologies fit the planned interventions. Under inputs, I would be looking at the skill base and diversity of the project team, whether or not the funding available was sufficient to complete the project satisfactorily and whether the capacity building was appropriate for planned activities and would be sufficient to provide sustainability for impact after the project finished. Under processes, my main questions would be the recognition and inclusiveness of partnerships, whether the roles and responsibilities were well-defined and whether there were any risks or negative consequences that I should be aware of. Finally under outputs, I would be interested in whether the communication methods and tools were adequate, whether planned networking included engagement of appropriate and needed stakeholders, whether the project was sufficiently aware if the enabling environment was conducive to the success of the project , where relevant – were links being made with policy makers, and whether scaling readiness was part of stakeholder engagement.
Section 4 of the Guidelines on the Key Steps in Evaluating Quality of Science in research for development proposes methods which are also relevant to development projects. These include review of documents, interviews, focus group discussions, social networking analysis, the Theory of Change and the use of rubrics to reduce subjectivity when using qualitative indictors. The use of rubrics is a cornerstone of the IDRC RQ+ Assessment Instrument.
United Kingdom
Jillian Lenne
Consultant
Independent consultant
Posted on 14/11/2023
I would like to contribute the following paper to the discussion. It was recently published by Outlook on Agriculture in a Special Issue on Agroecology:
“Measuring agroecology and its performance: An overview and critical discussion of existing tools and approaches”
Matthias S Geck, Mary Crossland and Christine Lamanna
Outlook on Agriculture 52:349-359
https://journals.sagepub.com/doi/full/10.1177/00307270231196309
Abstract
Agricultural and food systems (AFSs) are inherently multifunctional, representing a major driver for global crises but at the same time representing a huge potential for addressing multiple challenges simultaneously and contributing systemically to the achievement of sustainable development goals. Current performance metrics for AFS often fail to take this multifunc-tionality into account, focusing disproportionately on productivity and profitability, thereby excluding “externalities,” that is, key environmental and social values created by AFS. Agroecology is increasingly being recognized as a promising approach for AFS sustainability, due to its holistic and transformative nature. This growing interest in and commitment to agroecology by diverse actors implies a need for harmonized approaches to determine when a practice, project, investment, or policy can be considered agroecological, as well as approaches that ensure the multiple economic, environmental, and social values created by AFS are appropriately captured, hence creating a level playing field for comparing agroecology to alternatives. In this contribution to the special issue on agroecology, we present an overview of existing tools and frameworks for defining and measuring agroecology and its performance and critically discuss their limitations. We identify several deficiencies, including a shortage of approaches that allow for measuring agroecology and its performance on landscape and food system scale, and the use of standardized indicators for measuring agroecology integration, despite its context-specificity. These insights highlight the need for assessments focused on these overlooked scales and research on how best to reconcile the need for globally comparable approaches with assessing agroecology in a locally relevant manner. Lastly, we outline ongoing initiatives on behalf of the Agroecology Transformative Partnership that aim to overcome these shortcomings and offer a promising avenue for working toward harmonization of approaches. All readers are invited to contribute to these collaborative efforts in line with the agroecology principle of participation and co-creation of knowledge.