Open AI, Microsoft Azure, Gemini, Grok – the names of these artificial intelligences (AI) platforms are becoming increasingly common in the workplace. Organizations are adapting and integrating AI into their daily processes and increasing efficiency, and DevTech is no exception. DevTech is making strides to lead the use of AI in evaluations, while taking care to use it ethically and strategically.
As AI technologies continue to advance, the field of evaluation is poised for significant transformation. AI presents evaluators with a unique opportunity to enhance the efficiency, accuracy, and depth of their work, provided it is implemented responsibly and systematically across each stage of the evaluation lifecycle, from design to dissemination.
This article addresses AI’s strengths and potential for evaluation, its role at every stage of evaluations, examples of DevTech’s use of AI in evaluation, and how we safeguard ethics and standards while using AI.
AI’s Strengths in Evaluation
AI inherently excels at processing vast datasets in real time, often delivering analyses in seconds instead of hours or days.1 AI efficiently automates complex, time-consuming tasks such as data cleaning, labeling, integration, and anomaly detection. Moreover, when configured with consistent, objective criteria, AI systems can approach 99% reliability in advanced coding implementations, dramatically reducing human error.2
Beyond speed and scale, AI excels at uncovering patterns, trends, and correlations that might otherwise remain hidden, enabling evaluators to identify key insights and predict future outcomes with greater precision.3 In experimental and industry settings, generative AI coding tools have been shown to double task speed for developers,4 and in field trials across Microsoft, Accenture, and a Fortune 100 firm, these tools led to a 26% increase in completed tasks.
In a typical evaluation, an evaluator might review hundreds of background documents. While evaluators prioritize by categorizing and focusing on key documents, they may overlook less obvious but still valuable data. AI can help reduce human bias in collecting and coding this information.5
67AI in evaluation also customizes data collection, analysis (such as thematic analysis and statistical modeling), and reporting to fit project and client needs. AI, when utilized with a human-centered approach, can amplify, augment, empower, and enhance visuals in the virtual space.8 By integrating data from various sources, streamlining cleaning, and creating visual outputs such as dashboards, AI improves both findings and client experience, boosting impact.
AI Across the Evaluation Lifecycle
AI can add value at every phase of the evaluation process:
DevTech’s AI Use
Two notable examples of projects in which DevTech used AI for evaluation work are the Democracy, Human Rights, and Governance (DRG) Baseline Study and the Machine Learning for Peace Project. Both were created for the U.S. Department of State under the former U.S. Agency for International Development.
DevTech’s DRG Baseline Study employed a rigorous and intentional AI-assisted coding approach anchored in a detailed codebook specifically designed to reflect the objectives of USAID’s LCS Policy. This codebook operationalized the seven LCS principles and guided the AI in pre-classifying text, ensuring alignment with policy priorities. Rather than relying on generic pattern recognition, the AI was purposefully directed to identify relevant themes and passages, which were then refined and validated by expert analysts to capture critical context and nuance. This hybrid approach combined the efficiency and scalability of AI with the interpretive depth of human expertise, enabling rapid, comprehensive coding and analysis of 76 diverse program and policy documents within two weeks. Additionally, AI-assisted thematic analysis synthesized coded excerpts to uncover cross-cutting trends and gaps—such as limited attention to unintended consequences or mutuality—providing deep, actionable insights. Through this structured, policy-aligned methodology, DevTech transformed complex data into strategic knowledge, supporting evidence-based decision-making and advancing the Agency’s mission to strengthen local systems and foster sustainable development.
Under another work order, DevTech collaborated with the University of Pennsylvania’s DevLab@Penn team to support the MLP project, which harnesses machine learning and data analytics to strengthen democracy promotion and crisis response globally. MLP continuously scrapes and processes tens of millions of articles from over 300 local, regional, and international news sources in nearly 40 languages, tracking and forecasting major political events across more than 60 countries. In its quarterly forecasting reports, MLP integrates AI-generated summaries—using a GPT-4 model—to synthesize news articles covering detected shocks or major events, with human supervision ensuring accuracy and reliability. The project provides interactive digital tools, including the Civic Space Early Warning System (CSEWS), enabling activists, policymakers, and researchers to monitor civic space, assess foreign influence, and receive advanced warnings of potential crises, thus empowering timely, data-driven action.
Ensuring Quality, Ethics, and Accountability
As with any powerful technology, using AI in evaluation requires a thoughtful approach to quality assurance and ethics. Mitigating inherent biases in AI algorithms and their outputs is critical to maintaining fairness and objectivity across all evaluation phases. Evaluators must rigorously ensure compliance with evolving data protection regulations, especially when handling sensitive or personally identifiable information, irrespective of the AI platform’s location or public availability status. Transparency regarding how AI tools are integrated, coupled with clear documentation of methodologies and human oversight, is essential for building and maintaining trust with stakeholders. By prioritizing these ethical standards and robust accountability frameworks, evaluators can harness AI’s transformative power while upholding the core principles of rigor and responsible practice in the field.
A Future of Smarter, More Impactful Evaluations
The integration of AI heralds a future for evaluation that is profoundly smarter and more impactful. By automating routine tasks, enabling unprecedented data processing speeds, and revealing hidden insights across both quantitative and qualitative datasets, AI empowers evaluators to move beyond descriptive analysis to predictive modeling and real-time monitoring. This enhanced efficiency and analytical depth, coupled with the ability to personalize evaluations and visualize findings, will lead to more timely, evidence-based recommendations and ultimately, a greater and more sustainable positive change in programs and policies.
While AI-assisted functions within qualitative or quantitative analysis software are still in their nascent stages within the evaluation field, with evaluators actively navigating challenges such as ensuring nuanced contextual understanding, addressing algorithmic biases, and maintaining human oversight, this is undeniably the beginning of a transformative era for evaluation.
