Defining your approach

Purpose

The first step is to define the purpose(s) for the evaluation:

• Diagnostic (pre-course): Is the purpose to learn more about our students and their prior experiences (eg of particular approaches or tools to be used in the module) in order to inform your approach to supporting their study?  
• Formative (during the course): to enable us to reflect on the impact of the planned approaches during the delivery of the course in order to, where possible, make adjustments to activities as it progresses?
• Summative (post-course): to help us to review the impact of the course design and delivery methods on student learning, with a view to transferring lessons learned to future modules?

Focus

This will be followed by defining the focus for the evaluation and the key questions you would like the methods to address. For example:

• Is the alignment of activities to learning outcomes optimum? Is it understood by students?
• Is the staging and development appropriate? Is there adequate attention to familiarisation and community development? Is there a conducive atmosphere for learning?
• Are the instructions clear and is there adequate support (administrative, technical, pedagogic)?
• How effective is the design, and the balance and relationships between in-person and online elements?
• Are there any workload issues? Is there adequate variety and feedback? Are the tools used effective?

As a general rule, the evaluation focus should be aligned with the targeted learning outcomes for the course – what you set out to achieve with the initial design of the course - and consequently linked to the overarching course objectives.

Overall approach

Once you have defined the focus, you will need to consider your overall approach and how this will drive your choice of methods. This will be underpinned by your research philosophy in terms of what you believe can be reliably captured through evaluation methods, what is worth evaluating and why this is the case.

Broadly, this might involve the following approaches:

An outcome-based approach might focus on the measurement of how far students have met the targeted learning behaviours and objectives. This might involve measuring levels of engagement (eg time on task or number of visits to a VLE module site) and the effectiveness of the learning (eg performance in formative and summative assessment). 

An interpretive study might focus more on staff and student perceptions of study methods and how they are understood and valued. This could be achieved by inviting students to recount their experiences of learning in their own words, elaborating on the context of their learning and the link between formal and informal study methods. Attention would be drawn to affective and attitudinal variables (learners’ feelings and levels of motivation towards the approaches used), and the role of prior knowledge and experience in influencing their reception of the study methods. Staff could also be invited to provide their perceptions of students’ learning relative to previous performance or previous cohorts.

In case study evaluation these approaches can be combined to generate a multi-dimensional picture of the learning that has taken place. For example, qualitative techniques are used to capture data on students’ perceptions and feelings on how the course has unfolded, while quantitative techniques generate data on what students know and what they do.