P2PU is largely a grant-funded organization, which means that we encounter a lot of grant reporting requirements. Over the past few years, we’ve often encountered a tension as traditional notions of reporting, assessment, evaluation, and data collection come into conflict with our values of peer learning, community, and equity.
None of this is to say that we don’t think evaluation is important. We know that we are not perfect, so getting feedback is really important to our growth and evolution. As part of an Institute of Museum and Library Services grant with the Kansas City Public Library, we set out to design a new set of tools that both meet our reporting requirements and serve to reinforce – not contradict – our values.
We released our first attempt of these tools in April 2018, and the details are documented in our release notes on Github. Below, we’ve summarized a few themes that emerged for us during this process. Please let us know what you think!
Data collection is a missed opportunity for dialogue.
Sturla Bjerkaker and Judit Summers write in “Learning Democratically” that “the importance attached to learner feedback in the formal sector rests on a limited view of the student as consumer, encapsulated in the term ‘learner satisfaction’, rather than on a belief in the importance of dialogue.” With this in mind, we sought to design tools that connected people to other people in meaningful ways, rather than simply to accumulate data for ourselves or grant assessment sections. Two examples that we will be trying out:
- Welcome Committee: When a facilitator creates a learning circle, P2PU is often curious what their goals are and whether they have any concerns. Our first instinct was to create a Google Form that aggregated their responses somewhere, but this didn’t seem to align well with the values we try to instill in learning circles. Instead, we created a welcome committee email group, which anybody can join. This committee gets copied on the learning circle creation email that is sent to the facilitator, and prompts discussion with librarians from around the world. The “documentation” we end up with is a threaded discussion between learning circle facilitators helping to orient the new person to the community.
- Letter Writing Activity: At the end of a learning circle, there is a lot that we are curious about from the perspective of both learners and facilitators. In consultation with a group of facilitators, we recently introduced a letter writing activity by which learning circle facilitators are prompted in the final week of their meeting to write a group letter to a different learning circle that is about to start. The prompt will say something specific, like, “a public speaking learning circle in Detroit will begin in 2 weeks. Can you write a note welcoming them and telling them about your experience?” In this way, we are gathering information as a byproduct of dialogue, rather than at the expense of dialogue.
We value your feedback, therefore we will NOT keep it confidential.
In almost every survey I’ve ever taken, keeping my responses confidential has been synonymous with “valuing my feedback”. It is requisite that we create safe and secure channels of confidential dialogue. However, when we were thinking about gathering additional feedback from learners and facilitators, we realized that the way to best value folks’ feedback was to be open about their experiences – to not keep it confidential. After learning circles finish, participants and facilitators both get a short survey from P2PU with a few questions. Rather than use the responses to generate behind-the-scenes reports, we are upfront with participants about how their responses will be immediately put to use improving the community. A few examples:
- Course feedback: Learners and facilitators are both asked a few questions about the quality of the online course they used, whether there were any additional resources they found helpful, and who they might recommend this course for in the future. This feedback is aggregated and shared via the P2PU course page, so that future facilitators and participants can make smart course decisions informed by the community.
- Tips for future participants: We ask learners what tips they have for future learning circle participants. These responses are aggregated, reviewed, and then included in the welcome message that participants get when they sign up for a learning circle.
- What do you want to do now?: For many people, learning circles are a step towards something else. Prompting learners to name their goal, reflect on the extent to which they reached their goal, and discussing what they want to do now that they’ve finished the learning circle is nice for personal reflection, but it’s also an opportunity to work with our library partners to identify how they can better understand and support their patrons goals and desires.
Your feedback is just that – yours.
P2PU isn’t the only stakeholder who stands to benefit from feedback and reflection. Library systems and adult education programs are the ones who are ultimately providing the learning circle, and participants are investing time and energy into a program that ought to provide value back to them. Therefore, it’s important that we make sure data doesn’t get stuck within P2PU, but quickly is shared back with others who stand to benefit. We will try to do this in a few ways:
- Insights back to learners: As we begin collecting information in the coming weeks, we’ll work with learners to identify feedback that would be interesting for them to see – for instance, how their reactions to learning circles compare to other participants, or what people who took similar courses did after the learning circle.
- Reports for libraries: One way we can add more value to our community-based organization partners is by automating reporting requirements for them. Through the interventions mentioned above, we will generate a fair amount of insight during the learning circle. During the next few months, with support from the Siegel Family Endowment, we will work with a handful of library partners to determine how we can best utilize this information to better shape library programming and learn more about designing successful non-formal learning programs.
- Making research easier: We’re also improving our reporting tools so that we can collaborate with external researchers easier. Currently, we have to be very selective about when we take on a research collaboration because prompting feedback, collating and cleaning data, and modeling findings are very time intensive on our part. Through this project, we seek to greatly improve the ease with which we can open up our findings to those who want to conduct research.