Monitoring and Evaluation of Social Programs Kathy Lindert, World Bank Qualidade do Gasto Publico no Brasil June 26-27, 2003

Outline Objectives of M&E Conceptual Framework, example Monitoring (role, steps, etc.) Evaluation (role, steps, etc.) Dissemination and Uses Institutional Issues and Sustainability

Objectives of M&E Monitoring and evaluation can help policy makers in many ways: As planning and budgeting tool: to ensure effective allocations of government resources – planning, decision-making and prioritization As a management tool by revealing the performance of ongoing activities at the sector, program or project level, to inform decisions about expansion, modification, or elimination of programs depending on their effectiveness As an accountability mechanism, so that managers can be held accountable for the performance of their projects/programs and so that government can be held accountable for its performance To obtain and maintain political support, institutional resilience (even across administrations), and financing for successful activities

Marco Conceitual Longo prazo, ampla melhoria na sociedade Objetivos Metas (Impactos) Longo prazo, ampla melhoria na sociedade Objetivos Resultados Efeitos intermediários dos produtos sobre os clientes Produtos Produtos e serviços produzidos Atividades Tarefas empreendidos para transformar insumos em produtos Implementação Insumos Financeiros, humanos, e recursos materiais

Example: Secondary Education Goal (Impacts) Higher income levels; increase access to higher skill jobs Outcomes Increased skills; more employment opportunities Outputs Number of youths completing secondary school Activities Schooling Inputs $, facilities, teachers, materials

Monitoring

Monitoring of Implementation and Results Monitoring is a continuous process of collecting and analyzing information to compare how well a program is being implemented against expected results Monitoring tracks both implementation and results Enfoque tradicional: Monitoring of activities Definir atividades e responsabilidades Monitoring activities is important No entanto: É possível executar as atividades, cumprir as responsibilildades, mas não fazer progresso até as metas Question: how will you know when you have been successful? “Se não se reconhecer o sucesso, vai se valorizar o fracasso”

Main Types of Monitoring Impact Results Results Monitoring Outcome Output Implementation Activity Implementation Monitoring (enfoque tradicional) Input

Key Steps for Monitoring Acordo sobre os resultados a monitorar Seleção de indicadores de estos resultados Medição de “baseline” Eleição de metas quantitativas Gerenciamento com relação às metas

1. Acordo sobre os resultados a monitorar Outcomes make explicit the intended objectives of governmental action – the vision Consensus about these outcomes, and this vision, is important Participation in agreeing on outcomes is key Key stakeholders should be involved (e.g., government, civil society, donors, etc.) Desenhar para sucesso Evitar resultados impossíveis Motivar pessoal com uma visão inspiradora Criar linha de vista (“line of sight”) para todos que trabalham no sistema entre as suas ações e o resultado desejado “Se você não sabe para onde vai, todos os caminhos estão certos”

2. Seleção de indicadores Outcome indicators are not the same as outcomes Indicadores devem medir resultados mas tambem processos e productos Um resultado corresponderá típicamente a mais de um indicador Respondem à pergunta: “Cómo conheceremos o sucesso, se o vermos?”

2. Seleção de indicadores, cont. A good performance indicator must be: Clear (Precise and unambiguous) Relevant (Appropriate to subject at hand) Economic (Available at reasonable cost) Adequate (Must provide a sufficient basis to assess performance) Monitorable (Must be amenable to independent validation)

3. Establishing Baseline Data on Indicators A performance baseline is… Information (quantitative or qualitative) that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used to: Learn about recent levels and patterns of performance on the indicator; and to Gauge subsequent policy, program, or project performance

Data Sources May Be Primary or Secondary PRIMARY data are collected directly by the organization, for example, through surveys, direct observation, and interviews. SECONDARY data have been collected by someone else, initially for a purpose other than that of the program. Examples include survey data collected by another agency, a Demographic Health Survey, or data from a financial market. Secondary data often can save money in acquiring data, but be careful!

Dados de “Baseline” Métodos informais/menos estruturados Pesquisas de painel Entrevistas com interlocutores chaves Conversas com pessoas afetadas “Grupos de foco” Pesquisas únicas Análises dos registros officiais Observação de participantes Observação direta Census Entrevistas das comunidades Visitas de campo Experiências controladas Questionários Métodos informais/menos estruturados Métodos formais/mais estruturados

Practicality Are the data associated with the indicator practical? Ask whether… Quality data are currently available The data can be procured on a regular and timely basis Primary data collection, when necessary, is feasible and cost-effective

4. Metas Targets are the quantifiable levels of the indicators that a country wants to achieve at a given point in time Precondições Clara compreensão do passado recente Conhecimento dos recursos disponíveis Considerações políticas Talvez o mais importante sendo o custo de não cumprir a meta… Gerenciamento para sucesso Seja realista! Faixas? Nota: gestão por resultados não depende críticamente das metas

5. Monitoria e gestão por Metas “Abertura” faz parte do enfoque Desenvolver estratégias de divulgação para o público, internet, etc. Monitoramento deve ser contínuo Informações devem correr “verticalmente” e “horizontalmente” A nenhum ponto do sistema devem ser coletados dados sem ser analisados Cria incentivos em favor da qualidade dos dados

5. Monitoria Fidelidade Validade Pontualidade Metodologia estável e consistente Fidelidade Validade Pontualidade Medida clara e direta do desempenho relevante Dados atualizados e suficientemente frenqüente para influenciar decisões gerenciais

5. Monitoria: Freqüência Quanto mais observações tem, tanto mais os diagnósticos serão Seguros Sofisticados Tempo Melhorar acesso aos mercados Accesso ? Tempo Melhorar acesso aos mercados Accesso

Evaluation

O papel das avaliações Results-Based Evaluation An assessment of a planned, ongoing, or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process. Evaluation Addresses “Why” Questions: What caused the changes we are monitoring “How” Questions: What was the sequence or processes that led to successful (or not) outcomes “Compliance/ Accountability Questions”: Did the promised activities actually take place and as they were planned?

Uses of Evaluation To make resource decisions To re-think about the causes of a problem To identify issues around an emerging problem, i.e. children dropping out of school To identify alternatives To build support of public sector reform / innovation To help build consensus among stakeholders on how to respond to a problem

Evaluation Means Information on: Strategy Whether we are doing the right things Rationale/justification Client satisfaction Operation Whether we are doing things right Effectiveness in achieving expected outcomes Efficiency in optimizing resources Learning Whether there are better ways of doing it Alternatives Best practices Lessons learned

General Types of Evaluations & PERs Evaluations can occur at many levels: Evaluations of individual staff performance (including project managers) Evaluations of organizational units Evaluations of programs (focus of this presentation) Evaluations of sectors/policy areas (e.g., cross-sectoral issues, fiscal policy, overall education policy) Public expenditure reviews generally evaluate the broadest level (evaluations of sectors/policy areas), but they also draw on evaluations of programs

Types of Program Evaluations Process Implementation Evaluations: Provides detailed information on whether program is operating as intended (are we doing things right?) Provides continuous feedback to assist managers Provides detailed information to those interested in scaling up or replicating Identifies bottlenecks Cost-Benefit Evaluations: Calculates the cost of the intervention relative to results Are we doing things efficiently?

Types of Program Evaluations, cont. Impact Evaluations: Measure net effect of program to determine whether or not objectives are being achieved Tries to establish causality between activities and impacts Impacts include both those that were intended and those that were not Impact evaluations require a clearly established counter-factual: What would have happened without the project? Generally need to compare those “with” the project (beneficiaries, “treatment group”) and those “without” the project (“control group”) both before and after project

Stages of Evaluation Clarify objectives of evaluation Explore available information Design evaluation Form evaluation team (capacity?) Collect information (sample, questionnaire, pilot, data collection, entry, quality control, etc.) Analysis Reporting, disseminating, discussing results Using results: feedback to policies, budgets, planning

Dissemination and Uses of Information

Divulgação Informar sobre o estado dos projetos Prover índices sobre problemas Explicar problemas Criar oportunidades para considerar melhoras o alternativas Prover informação sobre tendências

Usando as Conclusões As a planning and budgeting tool: Formular/justificar orçamentos Ajudar alocação operacional de recursos As a management tool: Melhorar os serviços As a performance/accountability tool: Responder às demandas para “accountability” Motivar pessoal Monitorar fornecedores Provocar debate sobre problemas Reforçar confiança pública

Institutional Issues and Sustainability Demand for M&E is crucial Structural requirements (ex. national law that mandates M&E of all federal programs) Budgetary and planning links Papéis claros Papel formal para organismos de planejamento e finanças Credibilidade Sistema deve entregar boas e más notícias! Proteção dos gerentes contra conseqüências gravas Responsabilidade (“Accountability”) Papéis da sociedade civil, mídia… Capacidade técnica