56

September 2011

  • Send
  • Subscribe

ARTICLES

The second international external review of AQU Catalunya (2011-2012)

Josep Grifoll Saurí and Maria Giné Soca - Head of Quality Assesment Department and Head of Planning and Communication Unit, AQU Catalunya

It was just five years ago (2006-2007) that AQU Catalunya began its first international external review whereby an analysis was made of the Agency's compliance with the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).

AQU Catalunya passed the evaluation and thereby maintained its status as a full member of the European Association for Quality Assurance in Higher Education (ENQA), of which it was a founder member, and became one of the first agencies to be registered with the European Quality Assurance Register for Higher Education (EQAR)1.

What it means for the Agency to pass an international external review

The fact that the Agency passes an international external review implies recognition of:

  • the Agency's legitimacy to carry out external reviews of the quality of higher education. This means that it has an official status, with a clear mission that is linked to its work of review and evaluation, and that it acts in accordance with the principle of independence that is required of any external review.
  • the Agency's activities, which are implemented in accordance with the criteria of efficiency and efficacy established at the European level.
  • the Agency's sustainability to continue its function, especially with reference to its available resources.

What are the implications of this for higher education in Catalonia?

Firstly, it offers guarantees to the entire higher education community that it works in accordance with quality standards that are internationally accepted and recognised. The external review process that the Agency undergoes helps to improve its credibility, both in the immediate surroundings in which it acts and at the international level.

Successfully passing the external review facilitates agreements on the mutual recognition of reviews carried out by the Agency with other national and international agencies and institutions. The most obvious case is the Agency's ability, in accordance with Spanish regulations, to carry out the ex-ante accreditation of recognised university degree programmes and awards. The only agencies (at the Autonomous region level) in Spain to carry out such ex-ante accreditation activities are those that have passed the international external review, are full members of ENQA and that are registered with the EQAR . Out of the eleven agencies that currently operate in Spain, five are authorised to carry out evaluations for the ex-ante accreditation of degrees (the corresponding agencies for the regions of Castilla-León, Galicia, Andalusia and Catalonia, and the national agency, ANECA).

The second implication for the higher education system in Catalonia refers to the added value implied in having its own, internationally recognised review agency. This refers not only to the possibility of it being an active voice in defining and developing external quality assurance (QA) systems and presenting the Catalan viewpoint in the corresponding international forums, but also that it reinforces the profile of the higher education system in Catalonia as a dynamic group of universities as a whole that is at the forefront in terms of the implementation of measures for quality assurance and the enhancement of higher education programmes.

The international external review 2011-2012

In 2011, AQU Catalunya is again starting the process that will lead to the international external review of the Agency, which will be completed the following year.

What is evaluated?
International external reviews evaluate the compliance of quality assurance agencies with the ESG. More specifically, the attainment of these standards is assessed in terms of an agency's work in evaluating degree programmes and higher education institutions (programme and institutional review). However, they also offer the possibility other specific activities of QA agencies being evaluated. Availing itself of this right, AQU Catalunya, in the process in 2007, also underwent an external review of its work involving the assessment of university teaching staff, which is a particular feature of the Spanish higher education system, given that in the rest of Europe this is carried out by the universities themselves.

The ESG are divided into three sections:
  I. European standards and guidelines for internal quality assurance within higher education institutions (the internal quality systems of the universities)
 II. European standards and guidelines for the external quality assurance of higher education (external institutional and programme review)
III. European standards and guidelines for external quality assurance agencies (international external review of QA agencies)

The universities must comply with the standards in section 1, and QA agencies with sections II and III.

What the standards say
Section II: for the external review of the universities

  • Standard 1. Use of internal QA procedures: External QA procedures should take into account the effectiveness of the internal QA procedures described in section I of the European Standards and Guidelines. 
  • Standard 2. Development of external QA processes: The aims and objectives of quality assurance processes should be determined before the processes themselves are developed, by all those responsible (including higher education institutions) and should be published with a description of the procedures to be used.
  • Standard 3. Criteria for decisions: Any formal decisions made as a result of an external quality assurance activity should be based on explicit published criteria that are applied consistently.
  • Standard 4. Processes fit for purpose: All external quality assurance processes should be designed specifically to ensure their fitness to achieve the aims and objectives set for them.
  • Standard 5. Reporting: Reports should be published and should be written in a style, which is clear and readily accessible to its intended readership. Any decisions, commendations or recommendations contained in reports should be easy for a reader to find.
  • Standard 6. Follow-up procedures: Quality assurance processes which contain recommendations for action or which require a subsequent action plan, should have a predetermined follow-up procedure which is implemented consistently.
  • Standard 7. Periodic reviews: External quality assurance of institutions and/or programmes should be under- taken on a cyclical basis. The length of the cycle and the review procedures to be used should be clearly defined and published in advance.
  • Standard 8. System-wide analyses: Quality assurance agencies should produce from time to time summary reports describing and analysing the general findings of their reviews, evaluations, assessments etc.

Section III: for the international external review of QA agencies

  • Standard 1. Use of external quality assurance procedures for higher education: The external quality assurance of agencies should take into account the presence and effectiveness of the external quality assurance processes described in Part 2 of the European Standards and Guidelines.
  • Standard 2. Official status: Agencies should be formally recognised by competent public authorities in the European Higher Education Area as agencies with responsibilities for external quality assurance and should have an established legal basis. They should comply with any requirements of the legislative jurisdictions within which they operate.
  • Standard 3. Activities: Agencies should undertake external quality assurance activities (at institutional or programme level) on a regular basis.
  • Standard 4. Resources: Agencies should have adequate and proportional resources, both human and financial, to enable them to organise and run their external quality assurance process(es) in an effective and efficient manner, with appropriate provision for the development of their processes and procedures.
  • Standard 5. Mission statement: Agencies should have clear and explicit goals and objectives for their work, contained in a publicly available statement.
  • Standard 6. Independence: Agencies should be independent to the extent both that they have autonomous responsibility for their operations and that the conclusions and recommendations made in their reports cannot be influenced by third parties such as higher education institutions, ministries or other stakeholders.
  • Standard 7. External quality assurance criteria and processes used by the agencies: The processes, criteria and procedures used by agencies should be pre-defined and publicly available.
  • Standard 8. Accountability procedures: Agencies should have in place procedures for their own accountability.

Who carries out the review?
ENQA has been requested to coordinate the review of AQU Catalunya, and the European association has accepted, as it did in 2007.

ENQA is responsible for the selection and setting up of an international external review panel, which is independent from the agency being reviewed and consists of five members: one external quality assurance expert (from another QA agency), one representative of higher education institutions (appointed by the EUA), one European student representative (appointed by the ESU), one stakeholder member (for example, from professional sectors). The panel includes one secretary who is an expert on the external review of QA agencies.

How is the Agency evaluated?
The Agency produces a self-evaluation report in which it uses evidence to explain how it complies with each of the ESG standards. This report is reviewed by people who work or who have worked for the Agency, and also by senior officials from the competent public authorities and the Catalan university authorities. The objective of the review is to carry out a study in which, in addition to the Agency justifying its compliance with the standards, its strengths and weaknesses in relation to the standards are identified, and recommendations made to improve the running of AQU Catalunya.

The self-evaluation report is reviewed and approved by the Agency's Board of Management, which is composed of all the vice-chancellors (rectors) and chairpersons of the boards of trustees of the public universities in Catalonia, together with the majority of the vice-chancellors from the private universities.

The external review panel reviews the self-evaluation report and subsequently makes a site visit to the Agency, which lasts two or three days, during which time interviews are held with representatives from the different sections in the Agency, as well as those who have received services from the Agency (members of the governing and review bodies, representatives of the competent public authorities and the universities, the staff of AQU Catalunya, evaluators and the evaluated, representatives from the different institutions, etc.), with whom it checks the contents of the report, together with the evidence.

On the basis of this process, the panel draws up an external assessment report, in which it delivers its opinion regarding the Agency's degree of compliance with each of the standards: fully compliant (it acts entirely in accordance with the criterion/ESG standard), substantially compliant (to a large extent in accordance with the criterion/standard), partially compliant (some aspects or parts of the criterion/ESG standard are met while others are not, and the interpretation of the criterion/ESG standard is correct, but the manner of implementation is not effective enough) or non-compliant (the Agency fails to comply with the criterion/ESG standard). The panel also makes any appropriate enhancement proposals, the aim being for these to be introduced in time for the Agency's next review in five years time.

The Agency being evaluated has time to go over and clarify any aspects of the external report that it deems necessary. Following this, the panel issues a final report which is referred to the ENQA Board and EQAR for a ruling regarding whether the Agency will continue to be a full member of ENQA and registered with the EQAR.

What stage of the review process is AQU Catalunya in now?
AQU Catalunya has informed ENQA that it is starting the self-evaluation process using the guidelines given in the ESG, together with the enhancement proposals made by the external review panel in 2007.

The self-evaluation report on AQU Catalunya is to be produced during the last quarter of 2011 and, on completion, will be referred to the Agency's Board of Management for approval and subsequent public consultation.

It is envisaged that approval of the report will be given in March 2012. Following its approval, it will be referred to the members of the international external review panel designated by ENQA, in order for the site visit to AQU Catalunya to be organised in the spring of 2012.

All of the information on this process will be posted in a specific section on the Agency's website.

Behind its willingness to again undergo the international external review process for QA agencies, AQU Catalunya harbours a twofold desire. On the one hand, its desire to equal and, if possible, improve its rating in the first international review in 2007 and, on the other, to possess a valuable instrument for moving forward with the continuous enhancement of its activities from an international perspective.

1 Articles 24 and 25, Royal Decree 861/2010, 2 July, which amended Royal Decree 1393/2007, 29 October, whereby the regulations for recognised university qualifications were established. 

ENQA EQAR ISO

Generalitat de Catalunya

Via Laietana, 28, 5a planta 08003 Barcelona. Spain. Tel.: +34 93 268 89 50

© 2011 AQU Catalunya - Legal number B-21.910-2008