Using an Evaluation Rubric to Assess Systems Change in the Ecological Seed Sector in Canada

My name is Robyn McLean, and I’m the Principal Evaluator for Tapestry Evaluation and Strategy. 

As Jane Davidson describes, rubrics were initially developed in educational settings. I see them being extremely valuable for evaluating interventions with outcomes that are big-picture, hard to define – such as systems-change or community development initiatives.

I have applied this thinking in my evaluation work with the Bauta Family Initiative on Canadian Seed Security. It is a complex, systems-level initiative, aiming to build a national movement to conserve biodiversity, maintain public access to seed, promote sustainable production, and buffer our food system against the impacts of climate change. Developing an evaluation rubric has been extremely helpful for helping them first better understand what a resilient seed system looks like, and then to understand their contribution to building that system.

To do this, we identified 12 main outcomes we are trying to contribute to in the sector, and identified what different levels of those outcomes look like.

Why it will help:

  • When judging “big picture” concepts such as working towards a resilient seed system, evaluation rubrics will help us create a more holistic and comprehensive picture than individual indicators. [1] The staff for this initiative love how the rubric acts almost like a “report card” – allowing us to be very transparent about where we are at in comparison to where we want to be.
  • Doing the work to create the rubric has helped us become clear about where we are aiming to go – i.e., what we want the seed sector to look like. This can be difficult to do in systems-change level interventions.
  • Defining the changes we want to see is also helpful for communicating with stakeholders about what we are aiming for, and what the steps along the way will look like.
  • The rubric will help us plan the evaluation moving forward, by helping us focus on assessing the most important outcomes we want to shift in the system.

Hot Tip – How we have used the rubric:

  • Doing a baseline assessment of the seed sector before the initiative started, by making assessments of different outcomes on the rubric based on reviewing key background documents and data.
     
  • Using the rubric to guide regular staff reporting. We ask staff directly how their work in different regions and contexts contributes to the main outcomes we describe on the rubric. This helps ensure consistency across different regions, helps them keep their eye on the big picture, and helps them plan their work.
     
  • Using the rubric to help guide what data we collect. The rubric helps us focus on outcome data that assesses the main outcomes we have identified, including focusing on collecting data that helps us assess outcomes that we currently have less information on, and outcomes that are most important for leveraging change in the system.
     
  • Using the rubric to interpret all of the data we collect. Reporting on outcomes becomes easier, because we always start by describing what level we are at for each of our main outcomes, and then describe what evidence we used to make that assessment. It helps us be transparent about where we are at in relation to the system we want to see.

If you have any questions about how we developed or use this rubric - or your own stories of using a rubric -  we'd love to hear from you!