Connecting Engagement to Impact
The more information you have about the engagement patterns of network members or users of an online platform, the more tempting it is believe that these data alone can tell you everything you need to know. But, until you explore what type of engagement is valuable and why, and what kind of impact that engagement has on people, organizations and communities, your hypotheses about what actually drives outcomes remain untested.
Organizations often assess their network building efforts or technology interventions (or a combination of the two) to be able to come to more definitive conclusions about what works so that measures and indicators can be adjusted and an organization can learn from its experiences. With new technology that tracks people’s behavior (or even old technology like years of paper attendance records from different types of events) you can integrate actual behavior data on engagement over time with survey and other research to get a more comprehensive picture of how value and impact are created through engagement. You can also compare how engagement and other measures, such as number or type of connections in a social network, relate to impact.
With two recent projects, we were able to integrate engagement data with survey and other data to probe the value of different levels and types of engagement. The results offered insights into how impact was achieved and helped both organizations refine their network engagement strategies.
More Engagement on the Community Commons Means More Impact on Users
The Community Commons provides public access to thousands of meaningful data layers that allow mapping and reporting capabilities for people and organizations to explore community health and policy data interventions and best practices.
Data Collection and Analysis of Engagement – We worked with the Institute for People, Place and Possibility (IP3), the organization that stewards the Commons, to implement an online system to track user-centric data in a searchable, cloud-based relational database. This provided us with data to establish categories for a ladder of engagement based on engagement with core platform activities, such as building maps and reports, connecting to others, or reading tutorials to build capacity for using data.
Survey Data Collection on Outcomes – After a year of collecting platform data, we launched a user survey to explore what impact platform use and tool engagement had on users.
Results – Across key measures, the combined data showed greater impact for users who were more engaged. One of the core hypotheses in the Commons’ Theory of Action was that increased engagement with the platform’s tools would increase users’ knowledge, skills and capacity, a hypotheses that was supported by our research. A sample of the findings from this integrated analysis below.
Different Patterns of Engagement in Mozilla Science Lab Correspond with Different Views on Network Health and Outcomes
The Mozilla Science Lab is a network of researchers, developers, and librarians making research open and accessible and empowering open science leaders through fellowships, mentorships, and project-based learning.
Data Collection and Analysis of Engagement – In order to build a full database of people who had engaged with Science Lab over the years, we used event records, call attendance records, and GitHub data on code contributions and study group participation to create categories for both the level of engagement and the type of engagement of network members. This allowed us to compare diversity of participation – those people who participated in more than one way – to level of participation – those people who participated a specific number of times — as part of our analysis.
Survey Data Collection on Outcomes – As part of an existing cross-program survey conducted by the Mozilla Foundation, Mozilla Science participants were asked about their engagement in the networks that the Mozilla Foundation supports. Respondents were asked questions about the network’s health, and how they benefited from their participation in the network.
Results – We found that an individual’s levels of engagement and diversity of engagement correlated in slightly different ways with their reporting on network health and benefits (see results below for an example). Connecting the dots between patterns of engagement in a network and a range of network outcomes continues to be an important part of how we approach our network evaluation work.
