Evaluation capacity building in a rural Victorian community service organisation: A formative evaluation

Abstract

Community service organisations are increasingly required to report on outcomes and evaluate program delivery. While commonplace in clinical health settings, such work is novel to the community sector and can be challenging to undertake given resourcing and evaluation capacity constraints. These constraints are exacerbated for rural community service organisations that face additional resource pressures and lack organisational systems to support such work. This formative evaluation reports on an evaluation capacity building program within a rural place-based community service organisation in southwest Victoria, Australia. As part of the program, monitoring, evaluation, and learning pilots were implemented to support selected teams (N = 4 teams, N = 12 individuals) to increase confidence in outcomes measurement and reporting. Implementation strategies included training, creation of measurement frameworks, implementation of data collection tools, academic support, and emergence of evaluation champions. Evaluation data included administrative records and a staff survey. The Reach Effectiveness Adoption Implementation Maintenance framework guided evaluation. Results indicated that participants reported high awareness, beliefs, attitudes, knowledge, and skills in relation to monitoring, evaluation, and learning at the post-training/refinement period timepoint. However, several barriers limited implementation: reduced workforce capacity, prioritisation of client-driven or other work, leave or staff changes, additional responsibilities, waxing and waning engagement, wider program teams being unclear on the process and value of evaluation, and organisational barriers. These barriers led to a divergence from the planned outcome measures. Enablers to this work included physical presence to facilitate informal discussion, flexibility and openness to change, and communication to support staff. Learnings include the need to strengthen organisational evaluation culture, scaling-back the number of outcomes/data collection sources, and greater involvement with the wider staff cohort. These learnings provide a valuable foundation for other place-based community service organisations to implement evaluation capacity building strategies.