DIAL Evaluation Report (Open Educational Resources)

This report evaluates the progress of four projects within the JISC-funded UAL Project DIAL (Digital Integration into Arts Learning).   This two year project is coming up to the end of its first year of operation.  This report offers a formative evaluation, in line with JISC guidance on project evaluation. (DIAL Evaluation Report  Duna Sabri  October 201)

DIAL was conceived as a network of:

This evaluation (Report for year 1 - final) focused on 4 pilot project communities that were established in the first year:

Open educational resources community - led by Chris Follows and John Casey

Information literacies among library staff  - led by Jessica Crilly

Presentation skills among student entrepreneurs  - led by Laura North

 Online reflective practice for participants of the PG Cert in Academic Practice in Art,  Design and Communication - led by Lindsay Jordan

 The data sets for the evaluation of each of the above projects comprised:

1. An initial discussion about the aims of the project during January 2012 from which notes were produced by the researcher and in some in cases amended by the project leaders.
2. A follow up interview with project leaders in July 2012 which was recorded and transcribed.
3. Feedback from community members about the aspect of digital literacy that each project addressed.  This was collected and analysed by project leaders for the purposes of their own projects, and then, used for comparative purposes by the researcher.

The purpose of the initial discussions was to gain an understanding of project goals, the boundaries of the community to which they related, and the extent to which an assessment of skills and confidence levels had already been undertaken; and to identify a strategy for evaluation.  The follow-up interviews took the notes from the first discussion as their starting point and sought to understand what processes had been put in place to support development, what had been achieved, and what evidence was available in the form of feedback from community members.  To differing extents, in each case a transition had taken place in which project leaders’ conceptions of their goals had shifted and accordingly their processes for development, and the interviews often included a discussion about how evaluation strategies could be re-configured as well.

JISC evaluation guidance suggests the following questions to guide evaluation:

1. What have we done/built/achieved, to what quality and how efficiently?
2. What has been learned or confirmed through development activities?
3. How has the learning been acted on and fed back into decision-making?

The first two to of these questions are addressed in the course of this report, and it is intended that the discussion of this report, is a contribution to ‘feeding back into decision-making’.  The report incorporates comments from the project leaders as a first step in this process.  In addition to addressing these questions, this report attempts to place these four DIAL projects in a broader institutional context.

Initial observations about the four pilot projects

The aspirations for each of the projects have their roots in group leaders' reflections on their own roles and emerged from long-standing challenges that they wanted to tackle, and perhaps up to then had not had the time or resources to attend to them.  The issues they wanted to tackle could not be dealt with in a piece-meal day-to-day way but, they felt, would benefit from an injection of attention and resource that DIAL was to bring.  However, the resourcing of the projects, with one exception, relied exclusively on the goodwill and enthusiasm of the project leaders and, furthermore, the goodwill and enthusiasm that they were expected to generate among those that they sought to engage in their project community.

There was an affective dimension to all the projects that was related to the particular kind of digital literacy that they sought to develop.  The relationship between the affective and the technological varied from project to project.  For example, online reflection dealt with teachers' fear of learning in public, open access resources touched on the discomfort of making curricular resources public, and presentation skills development addressed anxieties relating to presenting oneself.

All of the projects were aiming to make some use of the relationships between different members of their target communities.  Some planned to get more literate/confident members to work with less literate/confident members, others aimed to bring together opposing views, staff working on similar courses from different colleges, or students from different cohorts.  The configurations varied depending on how the projects defined their particular hurdles.

How did the projects develop between January and July 2012?

In this section I take each project in turn and summarise the transition from initial aspirations to engagement with project communities.

Project 1: Open Educational Resources

The aim of this project is to expand the number of people who produce open educational resources (OER) and upload them to Arts Learning and Teaching Online (ALTO).  John and Chris, the project leaders, wanted to generate debate about OER that would address fears, misconceptions and aspirations in relation to OER.  They also aimed to review what had already been achieved in ALTO and ALTO UK with a view to informing their strategies for generating debate and awareness.

The first of these objectives was pursued through the conduct of focus groups with UAL staff in colleges.  These were conducted at CSM, LCF, Chelsea and Wimbledon.  Other significant discussions were held in other fora, notably at the Academic Leaders’ Forum, and on an impromptu basis in other settings.  The focus groups were exploratory and aimed to develop an understanding the range of meanings and responses that participants  attached to OER.

The data from these discussions resulted in the creation of a typology of awareness and engagement in OER.  This is intended for use as a diagnostic tool that allows individuals to see where they are in context – it allows them to recognise that there are different stages of engagement and that a period, sometimes a long period, of experimentation, is part of that.

There were also intentions to gain an overview through a systematic analysis of current users of ALTO but this was not carried out: partly through lack of time and possibly because it was not clear how such an analysis could be used.

In parallel with the planned activities in relation to this OER project two related developments promoted the systematic development of OER: the first was that Chris and John contributed to the development of a module on OER for UAL’s PG Cert; and the second was that Process Arts became a UAL service supported by the e-learning team.

Project 2: Information literacies among library staff

This project aimed to improve digital literacy among library staff as a precursor to developing library staff’s capacity to help students develop their digital information skills, such as, for example, evaluating online resources.  There were two inter-linked challenges facing this project: first to define the community of library staff that would participate in the project, and the second was to sustain the ambitious project activities with limited resources.  The project was originally led by David Bracegirdle as well as Jessia Crilly, but David withdrew when he took on new responsibilities.

Initially the project aimed to produce a resource inspired by the ‘twenty-three things’ format (such as that produced for researchers at the University of Huddersfield) and developed it into ‘unlimited things’.  Early on in the project, the two leaders identified that they did not know enough about the level and extent of digital literacy among library staff.  With this in mind, they initiated a discussion on Process-Arts which attracted a limited number of contributions – 11- but in excess of 900 viewings.  The discussion was about what tools individuals used, how they saw their levels of expertise, what they wanted to develop, and what they perceived to be the barriers.   In addition, they commissioned a survey of library staff at a Library Services Conference which drew 52 responses and appears to offer a reasonable overview.  At the time of writing it is not clear how representative this sample is of library staff as a whole.  Whilst some follow-up work may be needed to establish whether particular sub-groups may be under-represented, the survey seems at last to have established some characteristics of the target community for this project which might then form the basis of the design of ‘unlimited things’.

Project 3: Presentation skills among student entrepreneurs

The purpose of this project was firmly rooted in a survey of 160 students which was undertaken as part of a Teaching and Professional Development Fellowship.  The evidence from this survey suggested that there was both a need for art and design students to present in particular contexts and that they tended to find ‘nervewracking’ the experience of being filmed while speaking.  The idea of asking students to _develop a one-minute video pitch to present themselves or their project to potential clients, investors or a wider audience arose in collaboration with a lecturer, Chitra Buckley, who taught postgraduate students on the MA Fashion Entrepreneurship course. The project combined presentation skills development with digital literacy, both of which were relevant to the work of SEE.

The project defined its community as the students taking part in the MA course, but there was also a more general awareness of a wider group of students from other courses who could also benefit, and a sense that ways of working were being trialled and explored with a view to replicating equivalent experiences in other course contexts. With this in mind resources are being developed, including a video and workshop descriptions, in order to help other staff deliver similar workshops.

A series of three workshops was devised in collaboration with Chitra which six students attended. This revolved around alternating experiences of hands-on practice of recording one-minute pitches, and receiving peer feedback on the results.  Whilst there were numerous logistical difficulties, the project design included two data-sets of evaluation: the students’ perceptions of the experience and the students’ written feedback to each other about their presentations.  The first indicated that students quite often got over feelings of embarrassment, were intrigued by the process and threw themselves into the experimentation.   The second data-set indicated they had observed that the content and structure of the video pitches had improved.  It also revealed a great deal of potential for helping students to articulate and debate what would constitute a ‘good’ one-minute pitch in their particular context.

Project 4: Online reflective practice for participants of PG Cert in Academic Practice in Art, Design and Communication

This project shared with the presentation skills project an aspiration to help its community members to overcome an affective dimension of digital literacy.  The premise of this project was that ‘learning openly’ has intrinsic value for individuals and promoted collaboration and social learning.  At first the community was defined as all those engaging in online reflective practice who could be accessed through an already planned survey of blog-users.  The community definition shifted to the group of staff who were undertaking two core modules on the PG Cert in Academic Practice.  The line of questioning also shifted from being directly focused on the affective dimension of open online reflection to being more holistic, thus allowing Lindsay to situate the affective dimension in a broader context of other aspects of personal motivation and social contexts.

This group was surveyed about two tasks: the first was blogging monthly about set readings and providing peer feedback in small groups about other students’ blogs; the second was the experience of producing a short video encapsulating the conclusions of their teaching development projects.  The first survey attracted 24 responses and the second 17.  It is important to note that as well as in-depth feedback from these participants, Lindsay’s own interpretations of her own and the students’ experiences form an important part of the data for this project.

Anxieties about the openness of online reflection seemed to have dissipated and given way to larger concerns about such issues as technical know-how, the relative contributions of group members, and the difficulties of finding time to do the work.  In response to the question ‘how do you think this aspect of the course changed your perceptions of online learning?’ an interesting theme emerged relating to the degree of dissonance that participants perceived between the activity of online blogging and their learning.  It seemed that where online blogging did not relate to participants’ personal or professional goals, the blog was seen as ‘more of a repository than an aspect of online learning’.  For another participant, a move to a new job was closely related to the insights gained through the experience of blogging: for example this participant was able to situate blogging as a register of discourse somewhere between academic writing and informal communication.  Some recounted their efforts to overcome the dissonance between their experience of the technology and their learning.  What seemed to characterise these accounts was a close observation of the distinctiveness of the exercise: the potential for immediacy and the impact of delay between responses; and the challenge of finding one’s ‘voice’ in a new medium with its own conventions and structures.  There was also a range of experiences of the relationship between online and face-to-face communication within the small groups: face-to-face communication was most often perceived as a means of facilitating online communication, for some an essential pre-requisite.  Finally, there were explorations of the range of media that could be integrated into blog-posts, though this potential wasn’t fully explored by all who found it difficult to rely exclusively on the written word.

Some comparative features of the projects: language, institutional contexts and capacities to sustain development

The previous section has given an overview of the DIAL projects within their own terms of reference.  This section attempts to draw out some comparative features of the projects: similarities and variations which be suggestive of guidance for the initiation or development of other DIAL project communities; and may illuminate how the original aim of DIAL is working in practice.   These points are intended as a starting point for discussion among current and future DIAL participants rather than as a definitive analysis.  Furthermore, I have not sought to specify ‘recommendations’ as these can be more appropriately formulated as a result of discussions among DIAL participants.

1. The relationship between project leaders’ ‘day jobs’ and their DIAL community project varied.  Some were able to integrate projects into their existing work and into existing relationships with others. For example, Lindsay working with students on PG Cert was able to integrate the project both in terms of its substance and in terms of drawing on existing formal relationships with her students.  In contrast Laura reached across institutional boundaries to form a new relationship with a lecturer who shared a common interest, stepping both outside her job role and usual network of relationships.  Both are desirable models because they developed awareness and use of particular technologies and were conceptually transformative.  However, their different structures have different implications for resourcing and sustenance over the longer term.
2. For all the project leaders, as for the communities they sought to engage, prioritising the work of developing digital literacy was a perennial challenge.  The expectation that work on these projects could be sustained and systematic and whilst fitting alongside existing full-time jobs resulted in some inefficiencies.  There is some evidence that this expectation results in the exclusion of all but the most passionately committed to the development of digital literacies.  Beyond the four projects that this report focuses on, there have been other potential start-ups which fell by the wayside.  Worthwhile projects were identified in January, and planning meetings took place, commitments were undertake, only to find that subsequent correspondence went unanswered.
3. Related to the challenge set out in the previous point is another, that of sustaining a cumulative process of development within each community project and in DIAL as a whole.  Because time allocation to the project can be so piecemeal and scarce for individuals it was difficult, at times, to logically connect multiple data sets, or to make time to analyse one data set before going on to collect the next.  There is a need to follow through a process of inquiry: deciding on a question, collecting data, analysing and drawing conclusions from it before going on to collect more data or producing material.
4. An area of expertise that some project leaders wished to develop was that of analysing data.  The challenge they faced was twofold: first they lacked experience of systematic data analysis.  This meant that they had limited awareness of the strategies open to them for analysis and were unable to foresee their options at the point of choosing samples and data collection methods.  The second challenge was of being of being close to the data, often operating within the same context as their respondents, and therefore not necessarily problematizing the familiar.
5. There was necessarily a mix between opportunistic developmental processes and more systematic and strategic approaches.  It was not always clear whether the latter provided a context for the former or how the two related to each other.  To an uninformed outside observer initiatives can therefore appear somewhat incidental.
6. Allied to the previous point is a larger issue of structural presentation and communication beyond DIAL’s immediate interlocutors within UAL and beyond JISC.  There is a tendency to assume a shared understanding of terminology and of decisions to change use: for example ‘open educational resources’ have at different times been called ‘open education’ and ‘open resources’.  Alongside there are the acronyms SCORE and ALTO which is also referred to as a file-store.  As well as DIAL itself, each community project has its own title.  To the uninitiated it is difficult to work out both what the terminology means and how the different elements relate to each other.
7. It takes some time to realise that the proliferation of acronyms is rooted in the history of successive grant applications, each with their own title.  Whilst there is continuity and coherence in the developmental work that has taken place, the accumulation of acronyms gives the impression of disparate initiatives, unrelated to one another.
8. Finally, there was a growing feeling among project leaders that DIAL’s aim to produce ‘resources’ was worth exploring.  Among some there was a transition from seeing resources as handbooks or hand-outs towards seeing them as primarily residing within the new awareness and expertise of their communities, whilst also finding expression in certain online artefacts and practices.   This is reminiscent of the last of the common characteristics of the DIAL projects: their aspiration of making use of the differing levels of expertise within their target communities.  This remains an area of potential development in the forthcoming year.

Reference

JISC e-learning and project evaluation http://www.jisc.ac.uk/whatwedo/programmes/elearningcapital/evaluation.aspx