How to implement ROMA
ROMA is implemented with a mix of workshops, rapid reviews, detailed analysis and research, and time spent on reflection and learning. The RAPID team has found that, while workshops can be expensive – especially in terms of people’s time – they are a cost-effective way of building joint ownership of a project or programme when several partners are involved.
They also provide dedicated time to focus on the influencing project, beyond the day-to-day work environment. Individual pieces of analysis – from rapid reviews to well-resourced studies – can be done in preparation. But we have not found a good substitute for getting together in a room and grappling with the issues.
Much analysis can be done during a workshop, but it may be necessary to commission specific studies to gather more data and insight. Depending on time and resources, you must decide whether a basic analysis is sufficient or whether a separate, more formal study should be commissioned. The same is true for options for monitoring and evaluation (M&E).
The implementation team can collect and analyse a lot of data but this may not be enough. Time should be earmarked for more detailed enquiry or someone commissioned to conduct this.
How to use this guide
ROMA is a suite of tools that an organisation or team can use at any stage of their policy engagement process to:
- improve how they diagnose the problem,
- understand the types of impact their work could have on policy-making,
- set realistic objectives for policy influence,
- develop a plan to achieve those objectives,
- monitor and learn from the progress they are making,
- and reflect this learning back into their work.
This guide can be used in different ways by different people. Each problem requiring policy influence is unique, and the context in which you apply ROMA will be unique to that problem. So, ROMA is best used with a degree of flexibility: choose where to begin based on your priorities or your role. Who are you?
Team leaders
ROMA’s main message is to consider policy engagement as an integral part of the research or implementation process. Clearly, it is important that any attempts to engage or influence are based on robust evidence: for research projects and programmes in particular there is a tension between wanting to wait until the results are conclusive before communicating them, and wanting to engage during the life of the programme to be assured of a receptive audience when the results do emerge.
Using ROMA, you should be able to mix the two approaches, communicating throughout the life of your project so your stakeholders are aware of what may emerge and are able to consider possible responses.
Diagnosing the problem’s framework for diagnosing the issues you are facing will not only help structure your engaging and influencing strategies but also may help you refine what you do and whom you work with to bring about change.
Developing your engagement strategy’s workshop-based approach will help you decide whom to involve at what stage of the process to ensure your work is as cost-effective as possible.
From M&E to monitoring and learning's templates have been designed so the information you collect on your progress fits seamlessly into the way you manage your projects and programmes, and enables you to report effectively to donors. They will also ensure you can streamline the learning that happens (from both successful and unsuccessful influencing attempts) into your work planning and team management.
Monitoring and evaluation specialists
ROMA links monitoring to ongoing learning, not just to evaluation. In the face of complex problems, the ubiquitous phrase ‘M&E’ can be unhelpful: unpredictability means it is better to rely on ongoing monitoring to guide your work, rather than initial planning to guide your work.
Of course, evaluations are a key part of any project or programme, and the requisite information must be collected so that they are robust. ROMA suggests shifting the emphasis so more weight is given to ‘sense-making’ of monitoring information, fitting it into current management practices to ensure decisions about how to respond to an unpredictable situation are evidence-based and widely owned.
Diagnosing the problem shows why ROMA shifts the emphasis from ‘M&E’ to ‘M&L’ (monitoring and learning). The nature of complex problems means the ability to adapt is key, and this chapter helps you decide on what and with whom ongoing learning needs to occur.
Developing your engagement strategy will help you develop a monitoring and learning system that is broadly based and widely owned. Complex development problems are likely to involve many stakeholders, and the workshop tools suggested in this chapter will bring them together around a shared vision for the project or clarify where there is disagreement and divergence. It will ensure the learning process does not ride roughshod over those with weaker voices.
From M&E to monitoring and learning clarifies why we monitor, as the three pressures from donors (demonstrable impact, value for money and the need to address very complex development problems) can overwhelm monitoring systems. It may require a little initial effort to design the systems, but this will pay off in the long-term as information collection becomes more focused on supporting the decisions that will need to be taken and less on collecting it just in case it might be useful.
Researchers and practitioners
People in the thick of implementing a project or conducting a piece of research may sometimes find it hard to step back from their work to consider what else could be done, or how to do things differently. The danger is proceeding without testing assumptions about whom to collaborate with, what to do or how to communicate the evidence emerging from your work.
Diagnosing the problem will help you do just that, giving you a structured way of thinking about the issue you are working on, exploring the root causes and getting a clear diagnosis that may affect the way you conceive the issue and how you work on it. Clearly, work programmes cannot be thrown up in the air as soon as you notice a change in the wider context, but it is important to reflect on how those changes will affect the wider ramifications of what you are doing.
Developing your engagement strategy gives you a set of practical tools that will help bring your partners and other stakeholders into this assumption-checking process. Although the workshops are not designed specifically to do this, they offer a space for you to reflect – with your collaborators – on what you are doing and whether there is anything any of you could do more of, or differently, to promote sustainable change.
From M&E to monitoring and learning shows why monitoring should not be left just to the specialists: it is not enough to continue with the work programme you initially defined and hope an external evaluator will mark you highly. Monitoring needs to be a whole-team process, which everyone understands and to which everyone contributes.
Communications specialists
Good communications are a central pillar of the ROMA process. Achieving policy influence, particularly in complex policy environments, relies heavily on ensuring messages are well communicated in the right language to the appropriate audiences when they need them.
Diagnosing the problem reveals the importance of good communications and helps you focus on the key communication challenges. Communicating the nature of the problem you are facing is a key part of understanding what to do and how to do it; the more complex the problem, the more important effective, multi-directional communication is. Internal communications are as important as external communications, ensuring the project team shares this understanding will help to build links between the project and its wider stakeholders.
Developing your engagement strategy shows why it is important to ensure audiences include both people internal to the project as well as those outside. The chapter also helps you develop a strategy: ROMA’s workshop-based approach brings diverse stakeholders together. Maintaining good communications between them needs to happen for the life of your project or programme.
From M&E to monitoring and learning will help you build a communications strategy that is constantly informed by the latest monitoring evidence. The larger the project or programme, the more important it will be for specialist communicators to be included in monitoring processes to ensure learning is as widely shared as possible.
Policy-makers and civil servants
Policy-makers can use the ROMA toolkit to improve their own strategies for changing policy or practice within their departments or ministries. Where issues cut across departmental boundaries or where it is important to engage a variety of external stakeholders, the ROMA principles and toolkit have even more to offer.
Diagnosing the problem shows that, for large cross-cutting issues, diagnosing the challenges faced may give you a set of different ideas about whom to engage with and how.
Developing your engagement strategy sets out some useful tools you can use to map your own stakeholders, understanding where you might be able to use outsiders (such as researchers or non-government organisations) to help reinforce your position. It also helps you consider more closely what outcomes you might be seeking.
From M&E to monitoring and learning provides insights into how your own policies can be monitored and how to draw learning from them effectively. Discussing the principles and practices in this chapter with your delivery organisations will help you come to a shared understanding of how to prioritise what you need to monitor, with the resources you have, to give you a picture of how effective your policies and programmes are.
Donors and research commissioners
Donors recognise the challenges of working in complex situations and welcome multiple approaches to solving them. However, this does not always translate well into the nature of the impacts they seek, which can over-emphasise the delivery of outputs according to predefined plans and under-emphasise the importance of adaptation and learning.
Diagnosing the problem shows the importance of diagnosing the complexity of a policy issue, and the three key aspects that need to be considered when designing projects and programmes to bring about policy change. Not all issues are complex, but it is important to ensure the tools and techniques used to design, implement, monitor and learn from strategies for policy influence are appropriately tailored to the nature of the issue at hand.
Developing your engagement strategy shows why it is important to encourage those implementing projects and programmes to allow sufficient time for broad engagement throughout, not just at the beginning. This has resource implications: workshops are not cheap, but internal communications within messy partnerships are essential if everyone is to be able to contribute appropriately to broad-based sustainable development.
From M&E to monitoring and learning shows why it is important to focus on learning when the goal is to influence policy change around complex problems. Approaches to monitoring based on OM complement more traditional programme and project management techniques, helping unpick the behavioural assumptions that often weaken theories of change (particularly between output and outcome levels in a logframe).
To cope better with complexity, it would be helpful for donors to drop the term ‘M&E’ from the donor lexicon, changing the emphasis to M&L, and seeing evaluation as a separate issue.