Improving impact: Our approach to monitoring, evaluation and learning

Photo by Anupam Mahapatra on Unsplash

The Open Data Institute (ODI) and Sport England have embarked on the next stage of OpenActive, a major open data project to tackle inactivity. This phase of the initiative will focus more on enabling organisations to use OpenActive’s data infrastructure and making the initiative self-sustaining — something that Monitoring, Evaluation and Learning (MEL) will play a critical role in.

What does physical activity have to do with data infrastructure?

The Covid-19 pandemic has had a disproportionately negative impact on activity levels among marginalised groups — such as children and young people, older people, women, disabled people and culturally diverse communities. This has exacerbated existing inequalities — especially when considering the overwhelming evidence for the life benefits of being active, from childhood through to old age. These benefits deliver billions of pounds of value to our healthcare systems, society and economy every year.

A lack of access to high-quality open data in the sport and physical activity sector is a barrier to getting people active. OpenActive is a major open data initiative that enables the publishing of standardised open data on physical activities and promotes innovation to make it easier to find and book these opportunities online. While OpenActive has made progress over the past five years to standardise and publish this data, there is still work to do to ensure meaningful impact.

Recognising the limitations and challenges of OpenActive (detailed below), this next phase of the initiative will look to address some of the issues outlined in Sport England’s ‘Uniting the Movement’ strategy — like designing data infrastructure to support the specific needs of people who are less likely to be active, encouraging innovation in the sector, and creating a more inclusive sport and physical activity landscape.

Why MEL?

MEL frameworks ensure that an initiative is having its expected impact. They encourage continuous learning and improvement, provide an evidence base for decision-makers and funders, and boost accountability.

In April 2021, an external review of OpenActive, highlighted that the initiative needs clear ‘roles, processes and systems that allow for evidence-gathering, evaluation and communication across the initiative.’ In response to this, OpenActive is building MEL into both the work completed by the ODI and the wider work of the initiative. This will help with the longer-term sustainability of the initiative and combine data to demonstrate value and impact.

Fundamentally, MEL will provide an evidence base for whether OpenActive has met its objectives while ensuring that all key qualitative achievements and learning opportunities are captured.

Limitations and challenges in measuring impact to date

OpenActive focuses on sharing information about opportunities for sport and physical activity — and we’ve seen the number and variety of activities grow throughout the initiative. We can measure this growth quite clearly in the data, but it is only one aspect of the impact equation.

To understand how opening up data has enabled more people to get active, we need information on participation and take-up. This would be a colossal challenge in a broad sector like ours — collecting meaningful participation data from individual activity providers and grassroots clubs to large leisure operators. In time, as the coverage of opportunities reflected in OpenActive data feeds becomes more complete, we will be able to draw more robust conclusions about the relationships between the provision of opportunities, the use of open data and new tools, and health and wellbeing impacts.

To help overcome the challenges that we have faced measuring impact, we have outlined some key learnings that we have incorporated into our approach to MEL:

• Don’t underestimate the time and effort needed to work with and onboard an entire sector and to appreciate the varied approaches and methodologies within a sector. For example, large software platform providers have the technical know-how but need more lead up time for the benefits; startups may lack the technical knowledge but can implement quickly; non-governmental bodies have different needs to leisure operators.
• It is important to understand better the quality of data published and how the data is being used, to improve user experiences, encourage future publication, and explore updates to the specifications. In this phase, the team is developing a new data quality framework to achieve this.
• The challenge of getting the community involved was harder than expected. We found this has worked where there is mutual self-interest: there needs to be concrete benefits for all parties. For this, we have ensured getting an ideal sector representation in the steering committee, added a use case approach and activated the adoption engagement forums.

OpenActive’s approach to impact

Embedded MEL will be an essential element of OpenActive moving forwards and will support the core infrastructure to deliver social value and help address some of the challenges and limitations described above.

The steps to our MEL approach were:

1. Updating the theory of change and logic model
2. Creation of Key Performance Indicators (KPIs) and a MEL plan
3. Undertake monitoring and evaluation of the project
4. Facilitate learning and reflection
5. Evaluate the benefits and impact of the initiatives interventions

The OpenActive team has developed a logic model for this phase of the initiative that outlines the impact that it expects OpenActive to achieve.

OpenActive logic model

The logic model is a key feature of impact by design. It is the part of an MEL approach that seeks out social impact from the start and works backwards to identify the activities that we need to focus on to realise the intended impact:

1. The sports and physical activity sector is serviced by a truly independent initiative capable of meeting needs of industry, policy, and society
2. Reduce inequalities and barriers to entry in the sports and physical activity sector to improve the population’s health and wellbeing including those historically underserved
3. OpenActive becomes integral to national data infrastructure

You can find a simplified version of the logic model on the Open Data Institute blog.

Measuring impact through KPIs

In total, we have selected 40 KPIs to track and evaluate OpenActive’s progress, which we are making public as part of our open working commitment.

We’ve listed some of the KPIs and targets below:

• KPI: Governance documents inclusive of risk registers, TORs and a road map signed off by the Steering Committee and % level of consensus on the agreed model; Target: 3 and 100%
• KPI: Amount of funding secured to enable the transition to an independent entity; Target: £500k
• KPI: The implementation of the data quality reporting framework; Target: final version of the framework published
• KPI: The number of queries resolved by community members; Target: 70% of queries resolved
• KPI: Proportion of users of tutorial and guides leaving positive feedback (example, rating of 4 out of 5 or above on tutorial exit poll); Target: 70%
• KPI: The proportion of existing publishers demonstrating improvement against the framework; Target: 50%
• KPI: Number of OA activity types routinely published as open data; Target: 350
• KPI: Number of new metrics relating OA opportunities to socio-economic data. To demonstrate progress against our ambition to reach underrepresented groups. (Measures could include deprivation indices, ethnicity, socio-economic status, age, and gender); Target: 4
• KPI: Qualified potential use case communities; Target: 4
• KPI: Participatory and co-authored case studies demonstrating use and impact; Target: 6
• KPI: Number of meetings with high-profile government representatives; Target: 5
• KPI: Roll out of an embedded MEL function for the initiative; Target: framework implemented alongside a high-level Theory of Change

To see the full list of the KPIs, view the detailed KPI tracker.

Want to find out more?

You can find a summary of OpenActive’s approach to MEL alongside a simplified version of the logic model and the KPIs on the ODI website.

• Read the detailed KPI list
• Read the ODI blogpost on measuring the impact of OpenActive
• Read the Sport England blogpost on this phase of OpenActive
• Read the ODI blogpost on this phase of OpenActive

Or if you’d like to speak to one of the team, contact: hello@openactive.io