UNDG Blog

Posts tagged M&E

Uncategorized

The stories behind the numbers in Kivu

Results, results, results. The age old monitoring and evaluation question: how do you [actually] draw a connection between transformational changes in the lives of people and the development projects that aim to help them?

The hard part is that the traditional monitoring approach does not focus on measuring outcome indicators, a weakness corrected by a new monitoring method: SenseMaker Narrative Capture. This initiative focuses on transformational changes, and uses qualitative and quantitative methods and collects narratives shared by the beneficiary populations.

As head of the Monitoring and Evaluation unit in the UNDP Democratic Republic of Congo country office, I led the implementation of this new monitoring and evaluation approach in South Kivu. Overall, the project was designed to to support the stabilization of the South Kivu region, which has been part of a conflict since 1994 among several actors looking to expand their territories in the Great Lakes Region.

Overall we believe that strengthening community management of conflict resolution and social infrastructure will help reduce potential sources of tension, which will help displaced and refugee populations return and reintegration process.

Monitoring change with a participatory approach

Generally, we were interested to learn about the changes in the life of communities involved in this joint programme developed by UNDP, UNICEF and FAO and particularly, we wanted to capture people’s experiences and feelings around the Kivu conflict, peace-keeping efforts surrounding the conflict, and the reintegration experiences of displaced individuals.

For this purpose, we approached different organizations and community leaders involved in the peace process following the conflict in the region. Our idea was to seek for their support designing monitoring tools and instruments we were planning to use and, because they took part in this first phase of the process, the tools obtained added value to the project. This participatory approach ensured that the content of the tools and questionnaires was well aligned with the reality in the field. This reality check empowered us to move to the most challenging part of the process, the data collection.

Capturing the stories behind the data

During the the data collection process, more than a thousand community members shared with us their story about the conflict, the stabilization and the peace process.

On this process of capturing the stories, what mostly amazed us, beyond their content, was the storytellers’ feedback:

By sharing this story I realize how was my life before, during and after the conflict, I realize how bad a conflict can be, why it is important to live as a community, to bring our children up with a new mindset. I realize how the different actors: the local authority, the church, the national army, the self-defense groups were interacting to either maintain crisis situation or to improve the situation of the communities”.

Some of the participants also shared their positive feedback on the way the data collection was done:

“The way you designed the questionnaire without asking me to share my opinion but to tell my story was fantastic. I used to give my opinion for surveys conducted by other organizations but I was never able to look back on the conflict and all the horror, the death, the tears, the food insecurity that we had to face everyday.”

Through this methodology, we realized that assessing the situation helps the storytellers focus not only on their opinion but also on their past experience. That is why we believe that Sense@Maker is an interesting and relevant addition to the M&E exercise as it is a realistic tool based on the commitment and strong participation from the beneficiaries and we plan to use it to influence future programme design and implementation.

Among the findings, one pointed out that education is a top concern for the communities. According to the results, communities find education a key component to promote skills, knowledge and new employment opportunities. So we are currently studying how education can be used to achieve a deeper impact in shaping attitudes towards conflict resolution and expanding access to social services. We will keep you in the loop!

Share This Post

Uncategorized

If you can’t measure it, you can’t manage it

During my internship with the United Nations Development Operations Coordination Office (UNDOCO), we organized a virtual innovation fair devoted to Real Time or Frequent Monitoring.  With only coffee as an incentive, Rose Sherman and Mita Paramita from Brightfront Group did all the leg work.  They put together a panel of  four progressive tech companies and nonprofit organizations and set up a panel discussion that we broadcast to our UN colleagues across the world.   The webinar was a great low-cost way to share new innovations – part of the UNDG’s efforts to  apply new data collection solutions in contexts where circumstances change rapidly, and people’s own knowledge is the best resource.

  • First up was Spatial Collective, a Nairobi-based social enterprise that uses Geographic Information Systems for community development. Local community mobilizers are trained in various data collection techniques and undertake accurate near real-time field level data collection. Then, the data is visualized and made accessible on platforms connecting multiple stakeholders.
  • Gramvaani (meaning ‘voice of the village’) is a social tech company based in Delhi capturing rural voices through social media. In 2009, Gramvaani started Mobile Vaani, a social media platform equivalent to Facebook/YouTube/Twitter for rural areas with low literacy and low internet coverage. Mobile Vaani works with an Interactive Voice Response system that allows people to dial a number and leave a message about their community, or listen to content produced by others. Communities thus share information about each other and awareness of rights and entitlements increases. This increased awareness helps create demand for public services and, in turn, raises the accountability of local authorities. At the same time, the platform generates important data, for instance on service delivery, that can be used for decision making.
  • “1.4 billion people in the world are illiterate, one fifth of the world’s population, let alone the other 20 percent who can only read in their local language. How do you reach out and engage successfully with these populations?” That is the challenge, Question Box wants to answer by installing public callboxes, which enable community members to go for help and to provide feedback near their homes and place of business. The easy and cost-efficient methodology offers a new place to provide programme services and a new stream of localized feedback data. Using callboxes, information provided to communities can be centralized and limited human resources can serve more people.
  • No Unidentified Flying Objects but Unmanned Aerial VehiclesUAViators mission is to promote the safe, coordinated and effective use of Unmanned Aerial Vehicles (better known as drones) for data collection, payload delivery and communication services in a wide range of humanitarian and development settings. It is hoped that drones may help to deliver high resolution geographic information assisting crisis relief, for instance in areas destroyed by earthquakes.

Nuggets of wisdom for real time monitoring in development and humanitarian work:

Even though every environment is unique, there are still certain dos and don’ts that make success more likely when applying new innovations. Here are the main takeaways that I picked up from the conversation:

DOs:

  • Make sure to engage communities in open innovations. As much as we know we shouldn’t, many inventors come to a community with pre-developed products/solutions, based on the knowledge of “outside experts,” as Rose Schuman of Brightfront Group put it. Community members, as “inside experts,” possess a great deal of knowledge as well.
  • Think about the social embedding. The technology alone is not the solution, the whole approach around it often matters much more. Aaditeshwar Seth of Gramvaani explained us how they had to invent supporting tools for people’s engagement to spread their innovation, a task that required considerably more social than technical skills.
  • Build long-term relationships. Many efforts to collect data are one time actions without a sustainable concept in mind to create a relationship with the stakeholders. To sustainably engage your data source, you have to give back value. Question Box does so by giving people using their callboxes immediate help to handle their urgent problems.
  • Only layering different data taps the whole potential of new and untraditional data. The main benefit of new data technology is that it offers diverse and non-traditional data sources. Often, insights tremendously increase when traditional data sources are combined with new ones, for instance survey data with Participatory GIS[1] as Primož Kovačič of Spatial Collective
  • Don’t be afraid of the costs of real time monitoring innovations! Leaving no one behind doesn’t have to imply big costs. Basic technology can be acquired for a reasonable price and scaled up later. I was quite stunned when I heard from Andrew Schroeder of UAViators that you can acquire a highly functional drone like the DJI Phantom for less than $1000!

And now it is your turn: As Thomas Edison said – “There is a way to do it better – find it!” Many innovations don’t need to be created, they already exist. The challenge is to find those that can be used to solve our specific challenges.

[1] Participatory GIS implies making GIT available to disadvantaged groups in society in order to enhance their capacity in generating, managing, analysing and communicating spatial information.

Share This Post

UNDAF

Inclusive UN strategic planning: a survivor’s guide

Having survived the UNDAF process, I provide these reflections in hope that my personal experience and personal convictions will help you and your United Nations Country Team (UNCT). These comments reflect personal experience – and where experience failed to meet expectations, personal convictions. Most will be self-evident, yet not applicable everywhere; and all may be totally misconceived.

UNDAFs and Delivering As One: Tools to help the UN and its partners work better together

UNDAF is a process, not a document. The document will only be as good as the process that led to it. The document will soon be forgotten; the governance structure and work practices will durably transform how we work and how we are perceived.

In addition, in today’s rapidly evolving world, any analysis or programming framework will soon become outdated. The Standard Operating Procedures (SOPs) allow for documents to be revised by consensus when needed; smooth-running governance structures will make this easy. Sweat the process, keep the document short.

Delivering as One is a Swiss knife with an infinite list of options (‘SOPs’). Compared to earlier guidance, it represents an incremental step towards integration, not a revolution. Beyond the government formally asking for the application of the approach, there is no set minimum.

The government and the UNCT can pick and choose how far to go along each of the pillars. 100 percent of any one pillar may not fit a given situation. Choose an option that works for you and adds value.

Better to aim low and ratchet up the ambition once something is seen to work, rather than try for the moon, hit a snag and undermine the credibility of the entire effort.

 

Governance structures must be inclusive, participatory and transparent.

Inclusive UN strategic planning- a survivor’s guide

Not only do they underpin programming quality, they contribute to our accountability, which goes beyond the national authorities.

In line with the SOPs, I suggest:

  • One national steering committee;
  • Vertical thematic groups, one for each result area;
  • Horizontal groups for cross-cutting issues (gender, human rights, youth, disaster risk reduction etc.);
  • Monitoring and evaluation (M&E). All of the above to be co-chaired by a government and UN agency head;
  • Communications and Operations – unless included in the local version of Delivering As One, these last two do not require a government co-chair nor reporting to the national steering committee.

The national steering committee should be representative of all our partners at country level, keeping overall numbers reasonable – government, civil society, private sector, media, donors etc.

Thematic groups should include anyone interested in the subject matter. All those interested should be on a mailing list or shared workspace. Only some will show up at every meeting, but all must be members, receive all the information and be able to come when they feel like it.

The composition of the M&E Group is more technical. It should include the national statistics authorities, the national M&E association if there is one, M&E specialists in donor offices, etc.

With regard to their contribution to the UNDAF, the groups are accountable to the national steering committee. The UN co-chairs are accountable to the UNCT.

Collaboration, consultation and common work for a purpose

Thematic groups function as forums for information exchange, and contribute to project formulation and monitoring. They also drive the drafting of the UNDAF and feed regular (annual/semi-annual) reporting.

They should meet often and spend most of their time discussing development issues, not UNDAF bureaucratic requirements (best handled by a core group). People will come to meetings only if they gain something – information, understanding. Keep the meetings and the minutes short, the chairmanship lively and participative, and hold UN co-chairs up to their responsibility of representing the whole UN family, not their agency interests.

Thematic groups can also contribute to resource mobilization to fill gaps in the resource table. The support of effective and representative groups provides credibility to a funding request to a donor with regard to a project/programme included in UNDAF.

Jamaica Kids - UN Photo

Thematic groups should not duplicate existing sectoral coordination arrangements – ideally, there should be only one coordination mechanism per sector, chaired by the government, which, as a line-item activity, would meet the needs of UNDAF and Delivering As One. Piggy-back on existing arrangements.

The M&E group has essential accountability responsibilities: during UNDAF (and later on individual project/programme) formulation, it validates indicators and targets. During implementation, it provides the common format for thematic groups to report progress, and it validates the data they present.

Focus on development results, and where working together adds value

Joint programming is the aim; joint programmes are an optional tool – for use where and when clear benefits offset the administrative burden.

Results framework and M&E constitute the core of the UNDAF. Get that part right.

Convincing reporting demands sound M&E. Invest in M&E human resources at the agency and inter-agency (UNCT) level. Use national data systems – and strengthen them through a programme activity if need be.

Use indicators that show the UN improving people’s lives, not the number of conferences/meetings/seminars/workshops – or even laws passed.

As far as possible, avoid duplication by using UNDAF-wide reporting mechanisms to answer Agency-specific requirements – review meetings, results frameworks, reports etc. The UNDAF Annual Review report should cover most of the reporting requirements of the agencies/ funds/ programmes.

Ownership – theirs and ours. Place government counterparts front and centre in annual reviews. Let them own the UNDAF results and be the ones reporting on them to their national peers (consider offering prior training on the effective use of PowerPoint).

Consultants kill ownership – use consultants to facilitate the process and polish up the documents if need be; rely on UN staff and their partners for drafting, etc. UN staff’s participation in joint UN work must be reflected in their performance plan, and actual contributions highlighted in their performance appraisal. UN co-chairs can be asked to provide inputs.

Break the agency silos at work and at play: between UN staff, familiarity breeds sympathy. The more exposure to each other as individuals, not agency flag bearers, the better. Even short of a UN House, shared premises and facilities like a kitchen/cafeteria help staff meet informally. Joint work increases familiarity with each other’s working practices. Shared accountability towards shared clients builds solidarity. Inclusive email lists spread information and feed a sense of a shared identity.

This also applies within the UNCT: each agency representative should have a well-defined share of responsibility in the work of the whole, and be accountable for it to the whole (not to the Resident Coordinator!).

Leave comments; let’s talk!

Share This Post

Uncategorized

An app to monitor HIV services in Bangkok

Technology is enabling a group at risk of stigma and discrimination in Thailand to voice their concerns and make an impact.  iMonitor+ is a new application for a smart phone, tablet or computer. People living with HIV or seeking services can monitor and evaluate public HIV services at health centres in Thailand. Using iMonitor+ they can file complaints and even talk with service providers in real-time.

Technology as a participatory tool

The United Nations global survey for a better world, My World, sparked our interest in technology as a participatory tool for consulting marginalized groups. More than 70,000 Thais voiced their views about the post-2015 development agenda. Thai voters’ top priorities were “a good education”, “an honest and responsive government” and “better healthcare”. As part of the survey, consultations with marginalized populations across all provinces highlighted the need for more inclusive policy-making and people’s participation.

Adapting an app to meet a need in Thailand

The United Nations in Thailand analyzed the challenges raised by the survey and determined to seek innovative solutions by harnessing technology. Our aim: to bring policymakers, communities and individuals closer than ever for stronger accountability.  Next came the adaptation of a generic prototype to fit a specific need. UNAIDS adapted a new mobile application for smart phones, iMonitor+, to support HIV prevention and response in Thailand. The initial focus became a community at high risk: men who have sex with men (MSM).

iMonitor+ addresses immediate epidemic concerns

  • HIV prevalence among MSM is highest in Bangkok (31.3 percent in 2010, up from 19.2 percent in 2005);
  • 43,040 new HIV infections will occur between 2012 and 2016 in Thailand; 62 percent of them will be among MSM, female sex workers and their clients, and people who inject drugs;
  • Mobile phones are a great outreach tool for this group. The number of mobile phones in use in Thailand is impressive, with a 125 percent mobile penetration rate;
  • Research shows that nearly 40 percent of MSM use online sources for information.

What does the app do?

The app allows users to monitor and evaluate access to public health services, and to dialogue with service providers in real-time. Users are able to raise ‘alerts’ on quality of access to services and to report cases of stigma and discrimination. Data collected will support evidence-based advocacy to encourage policymakers to improve health programmes. It is a new tool for accountability.

Why is it working so well in Bangkok? Partnership.

This app is part of a city-level monitoring and evaluation strategy (M&E). It contributes to a multi-sectoral partnership built between governments, community-based civil society organizations, and the United Nations to leverage on advances in mobile technology to empower communities’ voices. The development of iMonitor+ for MSM created some new partnerships, too. This was the first time the Bangkok Metropolitan Administration partnered with Swing Foundation, a community-based organization committed to improve the quality of life for sex workers.

UNAIDS led the initiative with technical support from the Joint Team on HIV/AIDS and the UN Resident Coordinator’s Office. Together they took a step-by-step approach to first build the capacity of government and civil society counterparts to embrace new technology before  scaling up to the wider population.

Lessons learned

We launched the iMonitor+ initiative in late 2014. What have we learned?

  1. Adapt generic applications to fit the development needs of the country;
  2. Provide adequate training on its usage;
  3. Make the cost affordable;
  4. Ensure the technology is simple and can be used by those people who do not have access to latest technology;
  5. Prioritize joint planning through quality assurance forums that engage all relevant stakeholders and ensure national ownership and sustainability.

What’s next? Expand to new users & communities

We will continue to advocate for new Thai efforts to expand the use of iMonitor+ to new users and communities beyond the first initiative with MSM in Bangkok. Phayao, a northern province of Thailand, is already using iMonitor+, and ASEAN cities are considering its use for Getting to Zero, the UNAIDS strategy.

 

 

Share This Post