A bimonthly publication of AQU Catalunya

53

March 2011

During the last few months of 2010, the Agency presented the Framework for the ex-ante assessment, monitoring, modification and accreditation of recognised degrees (known by its Catalan acronym as the VSMA Framework) to all of the universities, both public and private, in Catalonia, the aim being to explain the characteristics of the framework and receive feedback from those involved in its implementation. All together, around nine hundred (900) people took part, the majority of which were vice-principals, deans, heads of department, academic coordinators of recognised degree programmes, and university quality assurance and academic management staff.

One of the sections that raised the most interest was that of monitoring, particularly through the use of quantitative indicators. Mention should be made of several of the ideas underlying this cornerstone of the programme enhancement process. Before doing so, however, three initial considerations are that:

• Monitoring is an academic act and so the ultimate responsibility for analysing the running of a degree programme, together with any enhancements, lies with the academic coordinators. To do so, they will also obviously require technical support and need to get feedback from the stakeholders.
• Monitoring must use all of the relevant available information, both public and private. The objective of monitoring is the continuous enhancement of the programme, and it will not be very useful (or beneficial) if it is merely seen as being a simple procedure to be complied with.
• Monitoring must be based on quantitative data that is subject to assessment, which can then lead to modifications and enhancements being made to the course.

The quantitative data, or indicators, used for the monitoring process need to help characterise and identify the teaching and learning process. They therefore need to conform to reality and describe the day-to-day situation that the university community has to deal with, and not some intangible situation. The universities themselves have thus been responsible, through the experimental monitoring programme, for defining selected indicators using figures and data extracted from the consistent and reliable UNeix system¹.

What indicators should be used? Those that describe the three stages in the teaching and learning process:

• In the process input, i.e. new-entry student and access to university, there are indicators that show:
  - The characteristics of the process of access to university, i.e. the number of places offered, student entry requirements and qualifications, etc., and, for Master's programmes, previous qualifications.
  - Personal characteristics (for example, full or part-time), family (parents' level of education, etc.) and social (need to travel, place/type of residence, etc.) of new entry students.
• For delivery of the programme, or, in other words, the human, physical and organisational resources provided by the university for education, as a function of its teaching model and which must take into account the specifications of the different stakeholders (society, student body, the governing body and the university community as a whole), there is:
  - The type of academic staff (ordinary, tenure-track, assistants, associated, etc.) that are involved in giving the degree (lecturing and teaching) and their qualifications, who are accredited by way of processes that recognise merits in teaching and research.
  - The number of hours of master classes, seminars, laboratory, tutorials, etc. received by the student and the group class size for each.
  - The use of classrooms, laboratories, libraries and other physical resources that support the learning environment.
  - Use of the virtual campus (volume of shared material, number of visits/hits, volume of downloads, etc.).
  - Availability, currentness and usability of information on the degree programme and evidence of the methods for assessment used (exams, study tasks, etc.).
  - External practice (placement, work experience) and mobility.
  - Career and professional guidance.
  - Student satisfaction regarding academic staff and their studies, and the satisfaction of academic staff with the teaching process and their use of specific and appropriate instruments.
• In the process output, i.e. the number of students that complete the programme, the number of years taken, the graduate labour market outcomes, the opinion of all of the stakeholders, etc., there is:
  - The educational outcomes that can be gauged according to different rates (achievement, drop-out, graduation, efficiency, etc.), which provide a complementary perspective of the delivery of each programme.
  - Personal outcomes, which include the acquisition of core competences that, amongst other things, can be measured through the final-year project or dissertation of Bachelor and Master's degrees and the self-assessment of these competences (graduate labour market outcomes survey).
  - Graduate labour market outcomes, which indicate the education-job match (learning outcomes in relation to professional requirement, and surveys on employer satisfaction.

Many of these indicators are already available separately through the UNEIX university information system (indicators 1, 2 3, 11 and 13 in the diagram), others are currently being developed (4, 9 and 10) and the rest will need to be developed gradually over the next few years in order to be user-friendly and sound.

Benchmarks are needed so that programmes and the universities themselves can be improved, and in order for there to be benchmarks, indicators are needed: 1) that are consolidated, homogeneous and sustainable over time; 2) that can be extended to the entire higher education system in Catalonia; 3) that can be contextualised and are comparable.

Benchmarks need to form an integral part of the management culture in universities, and they need to be clear and comprehensible to everybody. They ultimately need to help improve the universities and, at the same time, be more transparent. And there is also the other side of transparency to be taken into account, i.e. university autonomy. The universities will gain greater autonomy when they are capable of better explaining what they are capable of doing with the resources made available to them by society.