Creative New Zealand (CNZ) partnered with Culture Counts to conduct a year-long pilot study involving the Culture Counts Evaluation Platform. This pilot engaged seven organisations funded through CNZ’s Toi Tōtara Haemata Investment Programme and took place between January and December 2022.
The Pilot aimed to help organisations understand, quantify and demonstrate their own unique impact, as well as contribute to the development of nation-wide public-value dataset. The Pilot mirrored similar trials in Australia and the United Kingdom.
Throughout the pilot, Culture Counts provided CNZ with a Power BI dashboard for monitoring the data collection.
By the time the data collection had concluded, the seven organisations had distributed surveys for 68 different activities and collected 2,730 individual responses all together.
Each participating organisation received a summary report of the data they collected, which then contributed to the development of a published evaluation report that aggregated the data and reviewed the experiences of those participating in the program.
“It was incredibly easy to get customer data and we’d love to extend this more across our public programmes to compare with concert data. It gave an insight in to the way our audience responds.”Pilot Participant Read report
Requirements
Creative New Zealand, like many public funders of the cultural sector, seeks to support and encourage arts practice in a manner that derives public value. The CNZ Strategic Direction and Statement of Intent sets out this ambition to deliver: Dynamic and resilient New Zealand arts, valued in Aotearoa and internationally.
Culture Counts is a widely understood solution for defining, quantify and measuring public value. Uniquely for funding bodies, Culture Counts provides a solution for measuring the value of ‘funded activities’, as opposed to large-scale population-based research, which looks to determine the general public’s perception and attitude towards the arts. As such, Creative New Zealand was interested to understand the logistical, practical and operational means in which Culture Counts could be implemented with its funded cohort and to understand the benefits of the solution towards achieving its strategic goals.
In shaping the pilot, Culture Counts proposed a design that parallels similar research programs led by other funding bodies, including Arts Council England, Department of Local Government, Sports and Cultural Industries (DLGSC), and Create NSW. The program invited a selection of organisations, representative of the wider funded portfolio, and then work with them to employ the Culture Counts Evaluation Platform in their operational thinking and program delivery. Their implementation of the platform would seek to establish a data collection mechanism that could offer both Creative New Zealand and the funded organisations, insights into the outcomes they were achieving and then degree to which those outcomes aligned with their intended objectives.
“I really liked that we were surveying audiences on how the shows impacted them instead of just if they enjoyed them. This is valuable to us and our artists.” – Pilot Participant
Approach
Culture Counts, in collaboration with CNZ, designed the pilot study and its roll-out, which included the following steps:
- Strategic Alignment & Evaluation Planning: A review of key documents and policies from Creative New Zealand, including their strategic plan and strategic directions was compiled. A strategic alignment that aggregated these documents and aligned a list of proposed metrics from the Culture Counts Outcomes Framework that aligned to key themes and could measure relevant outcomes.
- Survey & Tools Development: Survey templates were created to reflect specific outcomes that aligned back to the strategic alignment, and could be utilised by participants to collect feedback.
- Evaluation Guidance & Documentation: A guide on best practice data collection, methods and processes was written by Culture Counts and supplied to participating organisations.
- Training & Subscription: Each participating organisation, as well as Creative New Zealand, were given access to the Culture Counts Evaluation Platform and provided with a training and onboarding session to discuss the purpose of the Pilot and how organisations could utilise it in their own operations.
- Power BI Dashboard: Data from all participating organisations were collated into one dashboard, which was monitored by Creative New Zealand throughout the Pilot and continuously updated by Culture Counts.
- Support: Ongoing support was provided to participating organisations throughout the Pilot, including new ways in which the tool could be utilised.
- Feedback: Updates were provided to CNZ throughout the programme. At the end of the Pilot, Culture Counts collected feedback from participating organisations via an anonymous survey.
As part of the Pilot, organisations were asked to distribute a survey template to their audiences, participants and/or stakeholders that included a range of demographic and outcome-based questions. These ‘core’ questions related back to the overall measurement aims of Creative New Zealand. The CNZ Evaluation Pilot survey template included four dimension statements that aligned with the three key strategic pillars of the Creative New Zealand Investment Programme; Diversity and reach, Dynamic arts, Resilient arts sector.
Organisations could then use the Culture Counts Evaluation Platform to customise and update the survey template, adding their own questions or metrics if they wished. This encouraged organisations to add additional outcome metrics to their surveys based on their own strategic and artistic goals, as well as empower them engage with evaluation practice in a more meaningful way.
To facilitate this, the strategic alignment document also contained a longer list of aligned dimension statements. Organisations could then elect to add these outcome measures to their evaluation plan and feed back in the CNZ dashboard. This process would allow CNZ to get an understanding of the other outcomes their funded organisations we’re looking to achieve.
Outcome
As part of the Pilot’s requirements, each participating organisation was asked to evaluate a minimum of two activities. The majority of participating organisations chose to evaluate more than this – with an average of ten evaluations per organisation. In respect to custom dimensions metrics, Social and Quality dimensions were the most popular choice for organisations, with organisations choosing to include an average of 8 dimensions per survey (4 of which were the ‘core’ dimensions).
When looking at aggregate results, the dimensions Concept and Diversity saw the highest levels of agreement, with 95% and 81% respectively.
Aggregate results were discussed with Creative New Zealand to help unpack the reasons for the results and to understand how ‘core’ dimensions are applied to cohorts. Feedback regarding the application of these ‘core’ dimensions was also provided by participants of the pilot.
When asking participant organisations about their experience:
- 100% said they had a positive overall experience
- 100% said they saw value of contributing to a nation-wide dataset
- 60% said they through the core dimensions were relevant to them
Each organisation was also provided with a custom Online Insights Report at the end of the Pilot. This gave each participant a tangible output that they could then use to communicate their value to funders and stakeholders, and further reinforced the value of organisations taking on responsibility for their own evaluation activity through a distributed evaluation mechanism like Culture Counts.
Image credit: Artspace Aotearoa
