Is our EduFinance Theory of Change true?
The Role of Monitoring, Evaluation & Learning (MEL)
“How do you know if EduFinance activities drive impact?"
"How do you validate your theory of change?”
“What evidence do you have that the program is achieving intended results?”
Basically, how do we know if more children are accessing quality education as a result of EduFinance’s work?
These are some of the most frequently asked questions Opportunity EduFinance hears when speaking with stakeholders, and they are great questions. We realize monitoring, evaluation and learning (MEL) is critical to understanding program outcomes and impact, and yet often the least understood, sometimes explained with unhelpful jargon and buried under layers of complexity.
In this blog, we hope to break down some of this complexity and explore the ways EduFinance approaches monitoring, evaluation and learning.
‘MEL’ starts with our Theory of Change
Our theory of change ties the activities which we do on a daily basis (inputs) to measurable outputs, ultimately leading to positive outcomes and our main impact objective – more children are learning and have access to quality education.
The components of the EduFinance program directly align with our theory of change:
- We partner with financial institutions to help launch and grow EduFinance portfolios (loans to school owners and parents).
- Once school owners have access to loans from our partners, we offer schools the opportunity to join the EduQuality program.
- Through EduQuality, school leaders learn to diagnose school quality, and implement school development plans prioritizing the areas they want to continue improving.
- School leaders are offered professional development workshops focused on skills necessary to run a sustainable quality school.
- Teacher mentors are offered professional development sessions to engage with best practice teaching techniques and learn skills to coach their peer teachers.
What role does MEL play in EduFinance’s work?
Monitoring considers whether EduFinance is doing what we said we would be doing.
This means collecting data that monitors program implementation for our work with Financial institutions:
- Financial institution partners engaged
- Technical assistance projects completed
- New EduFinance loan products developed and approved by senior leadership
And for our work with schools:
- School partners attending any teaching or professional development workshop
- Self-Assessment and School Development Plan submissions
This monitoring data gives us visibility into the performance of individual team members, activities across EduFinance teams, and level of engagement of our partners.
Evaluation considers whether EduFinance should be doing what we are doing.
This means collecting and analyzing data that gives us insights into whether our program activities are effectively supporting the assumptions we made in our theory of change, or if we need to adapt and adjust.
Data collection includes:
- School improvement loan and school fee loan portfolio analysis to evaluate if financial institutions are investing in schools and parents.
- Feedback surveys from financial institution staff trainings on what modules were most and least helpful to better understand additional support we can provide each partner as they begin lending to schools and parents.
- Partner school profile data, including school fees charged, to understand whether our partner schools are affordable to low-income families.
- Annual school quality self-assessment scores to analyze areas of greatest and least quality improvements each year.
Learning considers a broader range of research questions and what we can learn from our school and financial institution partners. Past studies have examined:
- Do school improvement loans, school fee loans and/ or the Education Quality program result in improved learning outcomes for children?
- Does the teacher mentorship and professional development program (TMPD) result in improved teaching practices and therefore improved learning outcomes?
- Does the education quality program result in a mindset change that leads to improved leadership and management of schools?
- In what ways do school improvement loan investments impact the community through job creation and economic investment?
- What is the relationship between school fee loans and future earnings potential for students?
- What are lessons learned and future strategies for EduFinance lending shared by our global network of financial institution partners?
EduFinance’s MEL framework is oriented around the components of our theory of change, considering the short/medium-term outcomes (i.e. assumptions) that are critical to our intended long-term impact (i.e. objectives).
Our framework also includes a scale of validation that considers the different levels of rigor that may be required to validate our theory of change. For example, monthly portfolio data reporting from our financial institution partners gives significant and quick visibility into capital access (a short-term outcome), while a multi-year randomized control trial will be necessary to fully validate whether the program drives statistically significant improvements in student learning outcomes. This is because of what must happen after a school improvement loan is made: the capital needs to be invested, classrooms or washrooms are built, more pupils are able to enrol in the school, driving further investment in quality. These improvements do not happen overnight but have been proven over long periods of time to be effective drivers of learning in other markets.
Who contributes to EduFinance’s Monitoring, Evaluation & Learning?
Almost everyone. Technology has expanded our ability to collect data directly from partners, in addition to our teams’ roles to collect critical data.
- M&E Specialist teams are hired and managed locally to carry out many of our assessment tools with partner schools.
- The EduFinance Technical Assistance Advisors provide staff trainees with a digital survey link to complete and submit feedback surveys.
- Education Specialists use digital tools to report on partner school engagement at meetings and workshops.
- Schools digitally submit self-reported data on basic school profiles, school quality (self-assessment), school development plans, professional development, and more.
- The Data & Business Intelligence team digitizes and maintains the tools in their various local languages, oversees data quality and mapping of data and analyzes data to be reported out to key audiences.
- EduFinance also establishes research partnerships to design special studies, collect new primary data, and share raw data sets for deeper analysis into specific areas of impact we want to better understand. Research partners include Radical Innovation for Social Change (RISC) at the University of Chicago, Bath Social & Development Research Limited and ZiziAfrique, based in Kenya.
What tools are used to collect data?
EduFinance has developed a range of customized tools aligned with our theory of change to collect qualitative and quantitative data. Examples include –
- School Profile – M&E Specialists conduct in-depth school profiling surveys with EduQuality partner schools, providing a full picture that includes the school’s finances, student enrollment and dropout, teaching staff and facilities. Tracking how this profile data changes for the same school over several years can provide insights into potential correlations between EduQuality and changes in the school.
- School Quality and Behaviour and School Leadership – This survey is designed to track changes in school leader perspectives and behavior over time, recognizing mindset shift is a key dependency of changes in the school. School directors respond to questions on management, policies, community engagement, and teaching and learning approaches.
- Classroom Observation – Classes are attended by M&E Specialists to observe and document the learning environment and the quality of teaching practice. This is the most challenging tool to execute and requires hours of training by each M&E Specialist.
- School Development Plan Assessment - A sample of schools participate in this in-depth survey to understand how the school developed its school development plan, the approach to implementation, and any evidence of improved quality directly linked to the plan implementation.
- Financial Institution Staff Training Feedback – Loan officers, branch managers and risk teams are asked to complete digital surveys after training workshops to share how each training module impacted their own skills. This provides Advisors an understanding of areas the staff may require additional support, and also highlights opportunities to improve modules and/or create additional training modules.
The above is only an illustrative example of the full suite of tools EduFinance has developed to collect primary data for MEL. The team has also built out significant database infrastructure to collect, store, manage and analyze this data.
Here are just a few of the topline stats…
There is nothing static about EduFinance’s MEL approach. We will continue cycles of data collection, analysis, learning, and program adaptations. Our aim is to ensure EduFinance’s model is best aligned to achieve our goal – more children learning and have access to quality education.
To learn more, check out our latest Key Insights and Studies, and subscribe to our mailing list for notification on upcoming publications.