From Ephemeral to Tangible: Building Evaluation Infrastructure with Pao Arts Center
Data+Soul and Pao Arts Center teams gather for a photo in front of an exhibit during project kickoff. Photo credit: Pao Arts Center
Background
As I pensively swayed with the crowd to melodies about loss and self-discovery, I could literally feel how musician Maddie Lam’s song was moving the audience. There I was, attending the performance of Stardust in a Dandelion at Pao Arts Center, an Asian American Pacific Islander serving arts and cultural center in Boston’s Chinatown. In my role of providing Evaluation Technical Assistance to Pao Arts Center, I came to the show to experience their work firsthand, and to support the staff in using butterfly-shaped sticky notes to capture audience reflections. Some stickies read, “Thinking about grief and quiet, solitude. Processing alone with others...” and “Better together!” in testament to the way the event helped people connect with themselves and each other. The stickies were making tangible something that was otherwise ephemeral.
The Challenges
Ultimately, I was at this performance because of a classic evaluation challenge: An organization feels it is having an impact serving its community, but doesn’t have effective tools or practices to fully document this impact or make data-informed decisions about scaling or shifting their work. Further, how can that organization learn in ways that are both culturally responsive and fit within organizational capacity? My job was to provide protected and structured time for staff to update their evaluation infrastructure and build internal capacity for data collection, analysis, and storytelling. Indeed, the butterfly sticky note idea came from Pao Arts Center staff; in my role I provided a method for coding responses, and summarizing them for storytelling in a way that staff could repeat in the future.
Data+Soul Approach
Our partnership began with our Story+Data mapping process, in which the full staff team collaboratively mapped the story of how their community would be closer to reaching their aspirations with the help of Pao Arts Center. From there, we updated their logic model and decided upon key metrics that could be collected during programs as well as through an annual survey campaign. Next, we began the focused work of creating and refining new workflows that staff could use to design evaluations for events and exhibitions; and developing a new bi-lingual online and paper survey that staff would administer, analyze, and learn from once a year. Along the way, my colleague Outlaw and I facilitated skill-building workshops for staff on data analysis; and, with support from myself and Pao Arts Center’s Arts Engagement Manager, each staff member led their own evaluation inquiry, processing and reporting on data they had collected earlier in the year. Staff involvement throughout was critical, as they grounded decisions about data collection and interpretation in the cultural and practical context of their community, ranging from tech-savvy millennials to older Chinese immigrants.
Pao Arts Center staff work through a qualitative data analysis exercise
Photo credit: Evan Kuras
Results
Did I mention that, in addition to our Technical Assistance partnership, our team also produced an Impact Report that wove together findings from program evaluations and the annual survey? Staff were enthusiastic to see their individual contributions in the report (including those butterfly-shaped sticky notes) among compelling data points from the annual survey. At the end of our partnership, staff shared feeling ownership of evaluation processes and confidence using data for program improvement and communications. While right-sizing evaluation for a team’s capacity is an ongoing process, as is ensuring that methods are culturally responsive, we ended the partnership in a very different place from where we started: from being able to feel the ephemeral impact of Pao Arts Center, to being able to document it tangibly, learn from it, and act on it.