A apresentação está carregando. Por favor, espere

A apresentação está carregando. Por favor, espere

1 Monitoring and Evaluation of Social Programs Kathy Lindert, World Bank Qualidade do Gasto Publico no Brasil June 26-27, 2003.

Apresentações semelhantes


Apresentação em tema: "1 Monitoring and Evaluation of Social Programs Kathy Lindert, World Bank Qualidade do Gasto Publico no Brasil June 26-27, 2003."— Transcrição da apresentação:

1 1 Monitoring and Evaluation of Social Programs Kathy Lindert, World Bank Qualidade do Gasto Publico no Brasil June 26-27, 2003

2 2 Outline Objectives of M&E Conceptual Framework, example Monitoring (role, steps, etc.) Evaluation (role, steps, etc.) Dissemination and Uses Institutional Issues and Sustainability

3 3 Objectives of M&E Monitoring and evaluation can help policy makers in many ways: As planning and budgeting tool: to ensure effective allocations of government resources – planning, decision-making and prioritization As a management tool by revealing the performance of ongoing activities at the sector, program or project level, to inform decisions about expansion, modification, or elimination of programs depending on their effectiveness As an accountability mechanism, so that managers can be held accountable for the performance of their projects/programs and so that government can be held accountable for its performance To obtain and maintain political support, institutional resilience (even across administrations), and financing for successful activities

4 4 Marco Conceitual Resultados Efeitos intermediários dos produtos sobre os clientes Produtos Produtos e serviços produzidos Atividades Tarefas empreendidos para transformar insumos em produtos Insumos Financeiros, humanos, e recursos materiais Metas (Impactos) Longo prazo, ampla melhoria na sociedade Implementação Objetivos

5 5 Example: Secondary Education Outcomes Increased skills; more employment opportunities Outputs Number of youths completing secondary school Activities Schooling Inputs $, facilities, teachers, materials Goal (Impacts) Higher income levels; increase access to higher skill jobs

6 6 Monitoring

7 7 Monitoring of Implementation and Results Monitoring is a continuous process of collecting and analyzing information to compare how well a program is being implemented against expected results Monitoring tracks both implementation and results Enfoque tradicional: Monitoring of activities Definir atividades e responsabilidades Monitoring activities is important No entanto: É poss í vel executar as atividades, cumprir as responsibilildades, mas não fazer progresso at é as metas Question: how will you know when you have been successful? Se não se reconhecer o sucesso, vai se valorizar o fracasso

8 8 Main Types of Monitoring OutputActivityInputOutcome Impact Results Monitoring Implementation Monitoring (enfoque tradicional) Implementation Results

9 9 Key Steps for Monitoring Acordo sobre os resultados a monitorar Sele ç ão de indicadores de estos resultados Medi ç ão de baseline Elei ç ão de metas quantitativas Gerenciamento com rela ç ão à s metas

10 10 1. Acordo sobre os resultados a monitorar Outcomes make explicit the intended objectives of governmental action – the vision Consensus about these outcomes, and this vision, is important Participation in agreeing on outcomes is key Key stakeholders should be involved (e.g., government, civil society, donors, etc.) Desenhar para sucesso Evitar resultados imposs í veis Motivar pessoal com uma visão inspiradora Criar linha de vista ( line of sight ) para todos que trabalham no sistema entre as suas a ç ões e o resultado desejado Se você não sabe para onde vai, todos os caminhos estão certos

11 11 2. Sele ç ão de indicadores Outcome indicators are not the same as outcomes Indicadores devem medir resultados mas tambem processos e productos Um resultado corresponder á t í picamente a mais de um indicador Respondem à pergunta: C ó mo conheceremos o sucesso, se o vermos?

12 12 2. Sele ç ão de indicadores, cont. A good performance indicator must be: Clear(Precise and unambiguous) Relevant(Appropriate to subject at hand) Economic(Available at reasonable cost) Adequate(Must provide a sufficient basis to assess performance) Monitorable(Must be amenable to independent validation)

13 13 3. Establishing Baseline Data on Indicators A performance baseline is … Information (quantitative or qualitative) that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used to: Learn about recent levels and patterns of performance on the indicator; and to Gauge subsequent policy, program, or project performance

14 14 Data Sources May Be Primary or Secondary PRIMARY data are collected directly by the organization, for example, through surveys, direct observation, and interviews. SECONDARY data have been collected by someone else, initially for a purpose other than that of the program. Examples include survey data collected by another agency, a Demographic Health Survey, or data from a financial market. Secondary data often can save money in acquiring data, but be careful!

15 15 Dados de Baseline Conversas com pessoas afetadas Análises dos registros officiais Métodos informais/menos estruturados Visitas de campo Entrevistas das comunidades Observação de participantes Entrevistas com interlocutores chaves Grupos de foco Pesquisas únicas Questionários Pesquisas de painel Experiências controladas Census Métodos formais/mais estruturados Observação direta

16 16 Practicality Are the data associated with the indicator practical? Ask whether … Quality data are currently available The data can be procured on a regular and timely basis Primary data collection, when necessary, is feasible and cost-effective

17 17 4. Metas Targets are the quantifiable levels of the indicators that a country wants to achieve at a given point in time Precondi ç ões Clara compreensão do passado recente Conhecimento dos recursos dispon í veis Considera ç ões pol í ticas Talvez o mais importante sendo o custo de não cumprir a meta … Gerenciamento para sucesso Seja realista! Faixas? Nota: gestão por resultados não depende cr í ticamente das metas

18 18 5. Monitoria e gestão por Metas Abertura faz parte do enfoque Desenvolver estrat é gias de divulga ç ão para o p ú blico, internet, etc. Monitoramento deve ser cont í nuo Informa ç ões devem correr verticalmente e horizontalmente A nenhum ponto do sistema devem ser coletados dados sem ser analisados Cria incentivos em favor da qualidade dos dados

19 19 5. Monitoria Fidelidade Validade Pontualidade Metodologia estável e consistente Medida clara e direta do desempenho relevante Dados atualizados e suficientemente frenqüente para influenciar decisões gerenciais

20 20 5. Monitoria: Freq ü ência Quanto mais observa ç ões tem, tanto mais os diagn ó sticos serão Seguros Sofisticados ? Tempo Melhorar acesso aos mercados Accesso Tempo Melhorar acesso aos mercados Accesso

21 21 Evaluation

22 22 O papel das avalia ç ões Results-Based Evaluation An assessment of a planned, ongoing, or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process. Evaluation Addresses Why Questions: What caused the changes we are monitoring How Questions: What was the sequence or processes that led to successful (or not) outcomes Compliance/ Accountability Questions: Did the promised activities actually take place and as they were planned?

23 23 Uses of Evaluation To make resource decisions To re-think about the causes of a problem To identify issues around an emerging problem, i.e. children dropping out of school To identify alternatives To build support of public sector reform / innovation To help build consensus among stakeholders on how to respond to a problem

24 24 Evaluation Means Information on: Strategy Whether we are doing the right things Rationale/justification Client satisfaction Operation Whether we are doing things right Effectiveness in achieving expected outcomes Efficiency in optimizing resources Learning Whether there are better ways of doing it Alternatives Best practices Lessons learned

25 25 General Types of Evaluations & PERs Evaluations can occur at many levels: Evaluations of individual staff performance (including project managers) Evaluations of organizational units Evaluations of programs (focus of this presentation) Evaluations of sectors/policy areas (e.g., cross- sectoral issues, fiscal policy, overall education policy) Public expenditure reviews generally evaluate the broadest level (evaluations of sectors/policy areas), but they also draw on evaluations of programs

26 26 Types of Program Evaluations Process Implementation Evaluations: Provides detailed information on whether program is operating as intended (are we doing things right?) Provides continuous feedback to assist managers Provides detailed information to those interested in scaling up or replicating Identifies bottlenecks Cost-Benefit Evaluations: Calculates the cost of the intervention relative to results Are we doing things efficiently?

27 27 Types of Program Evaluations, cont. Impact Evaluations: Measure net effect of program to determine whether or not objectives are being achieved Tries to establish causality between activities and impacts Impacts include both those that were intended and those that were not Impact evaluations require a clearly established counter-factual: What would have happened without the project? Generally need to compare those with the project (beneficiaries, treatment group ) and those without the project ( control group ) both before and after project

28 28 Stages of Evaluation Clarify objectives of evaluation Explore available information Design evaluation Form evaluation team (capacity?) Collect information (sample, questionnaire, pilot, data collection, entry, quality control, etc.) Analysis Reporting, disseminating, discussing results Using results: feedback to policies, budgets, planning

29 29 Dissemination and Uses of Information

30 30 Divulga ç ão Informar sobre o estado dos projetos Prover í ndices sobre problemas Explicar problemas Criar oportunidades para considerar melhoras o alternativas Prover informa ç ão sobre tendências

31 31 Usando as Conclusões As a planning and budgeting tool: Formular/justificar or ç amentos Ajudar aloca ç ão operacional de recursos As a management tool: Melhorar os servi ç os As a performance/accountability tool: Responder à s demandas para accountability Motivar pessoal Monitorar fornecedores Provocar debate sobre problemas Refor ç ar confian ç a p ú blica

32 32 Institutional Issues and Sustainability Demand for M&E is crucial Structural requirements (ex. national law that mandates M&E of all federal programs) Budgetary and planning links Pap é is claros Papel formal para organismos de planejamento e finan ç as Credibilidade Sistema deve entregar boas e m á s not í cias! Prote ç ão dos gerentes contra conseq ü ências gravas Responsabilidade ( Accountability ) Pap é is da sociedade civil, m í dia … Capacidade t é cnica


Carregar ppt "1 Monitoring and Evaluation of Social Programs Kathy Lindert, World Bank Qualidade do Gasto Publico no Brasil June 26-27, 2003."

Apresentações semelhantes


Anúncios Google