This blog was initially published by the Independent Office of Evaluation of IFAD (IOE) here.
I. Introduction
1. In 2024, the Independent Office of Evaluation of IFAD (IOE) launched a corporate-level evaluation (CLE) of IFAD’s institutional and operational performance under IFAD11 and IFAD12. The evaluation examined key dimensions of IFAD’s business model, including its financial architecture, operations and human resource management.
2. Data collection and analysis were conducted across multiple evidence blocks, including structured interviews with headquarters stakeholders, country case studies, past CLEs and thematic evaluations, country strategy and programme evaluations (CSPEs), and a growing corpus of strategic and operational reports.
3. To better structure the results emanating from these various evidence streams, and to facilitate meaningful and time-efficient analysis, IOE integrated generative AI into the evaluation process, an innovation aligned with IOE’s AI Strategy and IFAD’s data governance policy.
4. In simple terms, AI helped the evaluation teamwork sift through large volumes of information more efficiently, but without losing rigour or transparency. Equally importantly, it assisted the team tremendously in triangulation of data and information from a vast reservoir of varied sources.
II. What did we do and why?
5. AI was not used to replace evaluator judgment. Instead, it helped the team to organize large volumes of qualitative data, find evidence efficiently, and apply a more consistent approach for analysis of results across sources. It also made it easier to trace each finding back to its original source. Each AI workflow was designed to uphold the principles of IOE’s Evaluation Manual to ensure the quality, consistency, rigour and transparency of the evaluation, while complying with industries’ ethical norms, including safeguards on data privacy and protocols for human validation.
6. The main reason for using AI was scale. The CLE drew on multiple evidence blocks, including synthesis of nine corporate and thematic evaluations, two Multilateral Organisation Performance Assessment Network (MOPAN) assessments, 35 CSPEs, 62 country strategic opportunities programmes (COSOPs), 10 country case studies with input from more than 350 interviews, more than 90 key informant interviews with IFAD Management, stakeholders and Executive Board members, a 486-respondent e-survey, portfolio analysis, thematic deep dives, and an impact assessment. AI made it possible to work with a broader evidence base while keeping the analysis structured and transparent.
III. How did we do it?
7. For interviews, audio recordings were transcribed and converted into structured text, and extracted key points, timestamped quotations, and stakeholder attributions were mapped to sub-evaluation questions (Sub-EQs). The team also developed a chatbot trained on anonymized interview minutes. Evaluators could ask questions in plain language and receive answers grounded in the data, including quotes and links to the original transcripts. A similar approach was used for semi-structured interviews in country case studies, covering government counterparts, IFAD staff, project units, donors and private sector actors.
8. For documents, AI-based classification using predefined categories was used to screen large volumes of publicly available IFAD material, including CSPEs, COSOPs, MOPAN assessments, country strategy notes and Board documents. For example, more than 95 reports were reviewed for non-lending activities across 33 sub-dimensions, with accuracy generally ranging between 80 and 95 per cent. The same approach was used to examine operational issues such as procurement, disbursement, timeliness and budget management, as well as broader themes like transformational change. Each classified paragraph was tagged (by country, year and document type) and linked back to its source. This made it possible to filter and compare information easily, and reduced analysis time from weeks to days.
9. AI also helped with triangulation. It allowed the team to compare evidence across interviews, case studies and documents, ensuring that each finding was supported by multiple sources before being included in the analysis.
10. IOE applied robust validation mechanisms and safeguards to ensure credibility. AI outputs were treated strictly as analytical inputs rather than findings. Evaluators reviewed all outputs, checking them against original transcripts and documents, and confirming relevance before using them. All data were anonymized before processing, and analysis was conducted in secure environments in line with IFAD IOE policies and the UNEG Ethical Principles for Harnessing AI in United Nations Evaluations. AI results were also compared with human coding through spot checks and standard accuracy metrics. Prompts and coding rules were documented, and all outputs remained traceable to their original source. Infrastructure and systematic human oversight ensured transparency, credibility and traceability throughout the CLE.
IV. What did we gain and learn?
11. One of the clearest benefits was time. Tasks that used to take weeks (such as screening large numbers of documents) were completed in days, and finding specific evidence became much faster. The approach also improved consistency, as similar types of evidence were treated in the same way across countries and themes. The workflows developed can now be reused in future evaluations, saving time going forward.
12. Yet, not everything could be automated. For instance, complex and multi-dimensional concepts (such as transformational change or policy engagement) which merit a higher degree of contextual understanding, required expert interpretation on the part of the evaluation team. There were also technical challenges. Transcription errors, especially with specialized terminology or accents, and issues with scanned documents required manual correction. Designing effective prompts also took time and iteration.
13. Finally, while AI saved time in some areas, it required careful validation, including spot-checking and cross-checking across sources.
In conclusion, generative AI is not a silver bullet, but in the case of the IFAD11–12 CLE, it proved to be a practical ally, helping evaluators go beyond the hype to deliver faster, more consistent, and more traceable analysis while safeguarding rigour and human judgment.