In September 2016, USAID/Nigeria awarded DevTech Systems, Inc. the four-year Monitoring, Evaluation and Learning Activity (MEL). This contract aims to make USAID/Nigeria the premier practitioner in Monitoring and Evaluation (M&E) and Collaborating, Learning and Adapting (CLA). Out of the work of the MEL Activity, one of the main components and focus on the second half of the Activity is Field Based Monitoring (FBM).
**Background ** Traditional Third-Party Monitoring (TPM) – process of sampling and collecting monitoring data through field monitors and/or TPM data collection firms and conducting quality assurance to ensure reliable reporting to USAID. This has been conducted for years now in places like Yemen, Afghanistan, Iraq, in order to gather data in areas where USAID personnel are not permitted to travel.
The MEL Activity’s FBM platform pairs local knowledge with collection technology to analyze data and develop monitoring reports that give USAID/Nigeria greater insight to the success and challenges of their activities in hard to reach. The MEL Activity deploys FBM as not only for data collection, but as a project management tool used to collect and collate activity or project level information and verify field implementation in difficult to access areas, built on the lessons learned and experiences in other nonpermissive environments. We assist USAID Nigeria in evaluating programmatic assumptions and providing a better understanding of Nigeria’s operational context through providing near-real time information on hard to reach areas.
Process FBM verifies field implementation in difficult to access areas beginning with geographic areas identified as Level 3 and Level 4 by the Regional Security Officer (see Figure 1), along with high priority Mission Activities, as determined by MEL and it’s Contracting Officer Representative (COR), in coordination with the Program and Front Offices. Requests will come in from the USAID Nigeria Mission, followed by a planning session with MEL Staff and USAID Nigeria to determine which of the third-party firms is most appropriate for this assessment. A detailed planning with meeting is also held with the Implementing Partner (IP) upon arrival in the field. Once the monitors are in the field, the MEL FBM Coordinator serves as the information pivot point, ensuring the schedule is executed accurately, tracking all security and/or other incidents that cause an adjustment to the initial schedule, and tracking incoming reports. They are also responsible for following up with reports that are not received in line with the schedule.
As part of the assessment photos are taken of the sites including: with respondents, during observation and while the field monitors fill out the surveys. These photos are helpful for later when the firm has to write a summary report based on their field work. Once MEL receives a report from the field, the FBM Coordinator works with the MEL Monitoring and Evaluation (M&E) Specialists who review incoming reports (specific to their assigned technical office) and carry out the data quality assurance process to finalize the report.
Once finalized by MEL, the FBM Coordinator solicits the IP’s feedback on the report providing a specific timeline on which the IP must submit comments and/or its requested corrections to the report. MEL will add these comments verbatim into the Annex of the report and make changes to the report at the discretion of the MEL M&E Specialists. After the IP feedback period is concluded (or not if not received), MEL will submit the report to the USAID Agreement Officer’s Representative/COR to “reject/accept”. The specific format for and contents of the site visit report template have also been created, along with an FMB checklist.
It is important to stress that, like data quality assessments, third party monitoring is not an audit, but rather an opportunity for the IPs to report any challenges or limitations they are facing, context and implementation constraints, and to identify capacity building challenges that MEL may be able to fill, if deemed appropriate.
Data Sources MEL employs a modified utilization-focused approach with the goal in mind of strengthening the data that USAID employs to support decision-making and future programming. MEL focuses on what information the Mission wants to know, and keeps stakeholders involved in the process. In some cases, the data being collected is the same the IP would report on, in others, the inquiry is what the Mission wants to take a deeper look at.
Primary data sources utilized include –
- Key Informant Interviews – face to face interviews between field monitors and people at the site
- Focus Group Discussions– with a sample of IP activity beneficiaries
- Field Monitor Observations – what the members of the firms are able to observe while at the IP’s Activity site
- Digitally Produced Data – photos, GPS readings
Other secondary data sources utilized include: IP submitted source data such as annual reports, quarterly reports, our FBM protocols, and trackers and data sets provided.
The data collection process is a relatively straightforward one. Field data are entered into the mobile application (AskiaFace mobile), and the data is uploaded and received immediately (or as soon as the enumerator is back in network range). Quantitative data collected are coded where counts and frequencies were used for measurement, and qualitative data are translated and transcribed from the original language to English. Finally, there are three levels of data validation to ensure the highest quality data are reported – level one performed by the field monitors themselves, level two by the MEL Activity Specialists, and level three by the MEL Activity COP.
The data collection firms chosen for this work are experienced working in their often difficult geographic regions and, more importantly, are well versed in their technical area and they have demonstrated experience utilizing both verbal and written social science skills. We chose the Askia software suite based on scalability and ease of information upload while in the field. This software allows us to: script our own tools, design our own surveys, conduct the surveys on mobile devices, upload survey data, and quickly and easily analyze the results. Firms’ members have their own phones, and the MEL Activity advises that there’s no marking on the front or back of the phone. Firms’ enumerators can just delete the app if need be for security purposes
Challenges MEL’s Field Based Monitoring program has been highly successful in furthering the work of the activity, but it has still been challenging . Some of the biggest challenges identified from the pilot program and beyond include:
- Communications: Ensuring that scheduled site visits are communicated to the AOR/COR and IPs at least a week in advance to allow proper time for preparations. To promote this, the MEL team developed an email template to the AOR/COR along with a form letter addressed to the Chief of Party that identifies key information to help expedite the process.
- GIS Consistency: GIS coordinates were not uniformly collected, and a GIS plan was not part of our pilot plan. To resolve this, the MEL Senior GIS Specialist added coordinates guidance to our site visit reporting template and started including a question that notes the specific geo-located information that the Technical Office wants the team to collect on our site visit preparation checklist
- Reporting: The reporting and review timeframes for each FBM firm varied greatly. To mitigate that, we’ve conducted after action reviews providing feedback via the site visit template and verbally to each of the pilot firms.
- Visuals: The MEL team was receiving pictures of varied quality based on the FMB firm. To combat this, the MEL Activity added a discussion and overview of the USAID Graphics Standards Manual and the USAID Shooting Guide to the 4-day FMB training.
**Future Outlook ** As of July 2019 , it is estimated that we will complete a minimum of 10 site visits per month to reach an annual total of 120 site visits through the activity in FY2019. The FBM data collection firms will also receive additional refresher training as the activity moves into it’s final year.
