A quarterly publication of AQU Catalunya

77

April 2015

The accreditation of recognised university degree programmes is an important process from an administrative, institutional and educational, as well as social and individual point of view. It brings to an end, at least temporarily, a cycle of renewal of university curricula that began in 2007 with legislation introduced that same year (Royal Decree 1393/2007). Given that it is an administrative measure, i.e. the outcome of accreditation means that a university degree will either be listed with or excluded from the Spanish register of higher education institutions and degrees (RUCT), it also has consequences for the universities and their faculties, departments and schools as regards what is taught in a curriculum and, ultimately, for the society promoting and supporting the whole process.

Regardless of the outcome, however, accreditation is a learning process for the universities, schools, departments and faculties, as well as for the reviewers themselves. It is a learning process in that the review process facilitates the evaluation of an institution's activities relating to a given group of degree courses and qualifications, and it matches them against a series of quality criteria and the points of view of a group of teachers, students, professionals and QA experts who are unconnected. The evaluation for reaccreditation is a learning process insofar as a review is made of what has already occurred, but looking ahead to the future.

The accreditation of recognised university degrees and qualifications is also a group process. It involves various institutional stakeholders – the universities, a quality assurance agency (AQU Catalunya) and the public authorities – and a myriad of people who contribute their views about the unit (faculty, department or school) and the degree undergoing evaluation. These people include the members (the reviewers) of the external review teams who carry out the site visit to the institution and draw up and produce what is known as an external review report. The following considerations regarding the process of reaccrediting Master's degrees at Catalan universities are by someone who has held the position of panel moderator (chair) on various such review panels.

An analysis of the "how" of the accreditation procedure has led me to think of external review – and accreditation itself – as an iterative process of "successive approximation". If one proceeds under the assumption that error is "an inherent part of things being measured", the aim of accreditation would therefore be to "minimise any such error" and to underpin and give credibility to the work and, logically, the outcomes of the work of the reviewers. The iterative nature of the accreditation procedure designed by AQU essentially leads to a progressive alignment between two institutions, i.e. the faculty/university and AQU, on behalf of which the reviewers carry out their work.

In a first iteration, the unit (faculty, school or department) produces a self-assessment report on the degrees undergoing evaluation and provides a body of evidence that is analysed by the corresponding external review panel. The team of reviewers on the panel assesses the adequacy and sufficiency of the evidence and draws up a preliminary analysis report together with a written request, where applicable, for any necessary complementary information, which are then sent to the unit. A second iteration includes a preliminary site visit in which the external review panel's moderator and secretary come into direct contact with the unit and an initial examination is made of the strengths and areas for enhancement that have been identified so far. Regardless of whether there is a preliminary visit, a formal site visit is made to verify in situ delivery of the study programmes in the unit; the visit includes an exchange of ideas, insights and opinions and a match and alignment between the points of view of those who directly participate in a degree programme and those who have no connection with it.

A preliminary external review report is drawn up after the site visit and placed before the institution for its consideration, which according to the criteria of the report gives rise to a further alignment in the form of submissions, including observations, comments and/or disagreement over the contents of the report. The external review panel then accordingly takes into account the submissions and draws up a definitive external review report. This report is referred to an Accreditation Committee, which gives rise to another iteration in which a preliminary accreditation report is produced, which in turn is placed before the institution for its consideration. This is not to mention the internal coordination and discussions that take place between AQU and the external reviewers, and between committees.

The vision that I have of accreditation is one of an all-inclusive process that offers guarantees: it is subject to built-in deadlines, the participation of all the stakeholders, the aspect of public announcement, decisions based on documentary evidence, a two-pronged approach (AQU-unit; reviewers-AQU), revisions and submissions. From the time the self-assessment report is drawn up, there is therefore a lengthy process before the final accreditation report is issued, so if it's any consolation, similar processes carried out in other countries by either national or supranational agencies take a similar amount of time. Being able to reduce the time of each iteration and the overall total time without undermining the safeguards in the process is currently a major challenge for everybody.

Participation in a process that offers such guarantees raises certain issues and is also an important challenge for the members of the review panels. The first such challenge is without doubt the matter of discriminating between different levels of quality. Discrimination means fine screening and determining the degree to which a programme complies with a given standard; in other words, the difference between "accreditation on-track-to-excellence", "accreditation" and "accreditation with prescriptions". In order to distinguish between these different degrees of compliance one has to look at all of the details and carefully examine all of the evidence. Achieving a good level of discrimination is important in order for the analysis and recommendations that are referred back to the institution to be valid, i.e. that they are useful in terms of improvement and enhancement, and that those who read them do not get the impression that they represent assertions that could have been made without so many procedural complications (red tape).

A second challenge for the reviewers relates to the consistency of programme assessment. Firstly, there needs to be overall consistency for an across-the-board application of the evaluation criteria to each and every dimension of the programmes being analysed. Secondly, vertical configuration of the criteria calls for a linear coherence so that, for example, there is no contradiction between statements about teaching staff and the assessment of learning outcomes. Thirdly, the wording and arguments set out in the reports need to be coherent and logical, so that a clear idea comes across of the reviewers' point of view with regard to either a favourable or unfavourable outcome for the accreditation procedure.

Perhaps the biggest challenge for the reviewers however is the credibility of their assessments in the eyes of the universities. In other words, the extent to which institutions feel they are sufficiently recognised in the mirror we hold up in front of them in the form of external review and accreditation reports. Achieving credibility is important in that, if accreditation is not granted, doubt could be cast on the assessment of programmes set out in evaluation reports and the recommendations made. In addition to those stemming from the actual outcome of accreditation, the usefulness of the reports for an institution can be seen in terms of the extent to which issues raised and recommendations proposed in the reports actually lead to continuous programme enhancement. This concern of the reviewers is understandable in light of the fact that a review or evaluation is ultimately only valid insofar as the consequences and outcomes of the review are valid.

The wording of the final report is one final challenge that reviewers have to deal with. Clear interpretation requires that their arguments be comprehensibly set out for the institution and in a way that gives weight to their assessment and is logical to the reader.

In order to meet the challenges that face reviewers, there are certain actions that institutions can carry out to help improve assessment by reviewers and ultimately the accreditation procedure that assessment refers to. Given that the coherence of the reviewers' appraisals is based to a great degree on the quality of the evidence that has to be examined, good monitoring and the internal review of study programmes could be said to be the best way of assuring the quality of review and evaluation. To this end, the self-assessment reports need to offer not just a description of the facts, data on satisfaction and indicators, but also an interpretation of these. This would enable the reviews to find out and understand the point of view of those who are most directly involved with the programme as regards, for example, a change in a particular piece of information; the relation between a piece of information and its background; a measure that has been adopted and its consequences, etc. Reviewers are sometimes only made aware of the institution's point of view in the submissions stage, in which the comments and interpretation put forward and presented by the institution are on occasion much clearer and more explicit than the actual self-assessment report.

As I said at the beginning, accreditation is also a learning process for those who carry out the evaluation (the reviewers). It is therefore down to us to make our appraisals more stringent and improve the way we put across our arguments in a way that helps to better guide academic coordinators as far as the next step to be made with the programme's design and/or delivery is concerned. It is at this juncture where positive synergies can be established between the learning process of reviewers and that of the institutions themselves.