November 2014
The commitment of AQU Catalunya to improving the effectiveness and efficiency of quality assurance takes the form of regular meta-evaluation actions, which act as feedback and deliberative mechanisms for QA development and improvement.
In the context of the accreditation of recognised First and Second Cycle (undergraduate and Master's level) university degree programmes, on 17 September AQU Catalunya organised a meta-evaluation session, based on the experience gained from the nine external site visits carried out so far in relation to programme accreditation between April-July 2014. Stakeholders in accreditation were widely represented among the approximately fifty participants in the session and included programme coordinators, heads of QA units and offices, and members of the external review panels.
The proceedings of the session were organised according to:
An analysis is given below of the quality of the evidence provided by the faculties and schools in the self-evaluation reports, a key factor in ensuring the quality of the accreditation process in general and that has to do with the aspect dealt with in point two of the session on 17 September.
The purpose of this analysis is to highlight the main areas for improvement identified by the external review panels. For the analysis, use was made of the review reports on the self-assessment report and evidence, which are the reports drawn up by the external review panels prior to the site visit and where the faculty or school is requested, where applicable, to provide new documentation or additional information.
The requests by the external review panels for further or additional information in relation to self-evaluation reports submitted during the first half of 2014 were as follows:
1. Quality of the study programme
With regard to this first standard, a request for information on, and an assessment of, the profile of new entry students to be included in the self-evaluation report was made on six occasions. This information can be summarised as:
To a lesser degree (two occasions) information was requested regarding teaching coordination mechanisms.
2. Relevance of the public information
In relation to the second standard, in the majority of the applications (seven) there were shortcomings detected in the information provided on the website of either the institution or the study programme.
Requests were made for:
On one occasion documentation on the procedures in the internal quality assurance system dealing with public information, the gathering of information and accountability was also requested.
3. Efficacy of the study programme's internal quality assurance system
The third standard, referring to the internal quality assurance system and its efficacy, resulted in a large number of requests for the information provided to be either supplemented or modified.
As for the instruments for gathering information regarding stakeholder satisfaction, eight requests were made for the following to be added to the self-evaluation report:
The request for improvements to be made to the quantitative information on programme delivery was also made on eight occasions, in the form of:
With regard to the review and revision of the internal quality assurance system, detailed information on the implementation of procedures and the time limits/deadlines for implementation was requested on four occasions.
Lastly, the plans for, and monitoring of, improvements and corrective action were commented by the external review panels on nine occasions. The main requests were:
4. The suitability (qualifications and level of competence) of teaching staff on the study programme
Regarding standard 4, in reference to academic staff, the external review panels in particular called for more extensive qualitative and quantitative information on the experience of teaching and research staff in teaching, research and at professional level.
Secondly, seven requests were made in connection with the suitability and level of competence of academic staff on study programmes, with:
Requests were also made for new information regarding the matter of the institutional support received by academic staff, more specifically:
5. Efficacy of learning support systems
Relatively speaking, standard 5 is the one where fewer requests were made for more extensive documentation. Requests were made on three occasions for an analysis and comments on the workings of the plan for interactive tutorials to be included in the self-evaluation report.
To a lesser degree, the external review panels requested indicators on the level of satisfaction with academic tutorials, satisfaction indicators on careers and professional guidance, and documentation from the internal quality assurance system on student support and guidance.
6. Quality of programme learning outcomes
Requests for new information were made in relation to two sections of standard 6. The highest number (seven occasions) of requests were for examples of the students' achievements, more specifically:
Requests in the second section were for the trends, together with an evaluative analysis, in the indicators of both the performance of graduates of the study programme and their transition to the labour market.
Conclusion
The overall assessment by the external review panels of the quality of the self-evaluation reports and other supporting evidence connected with the site visits is, in general, quite positive, bearing in mind in particular that these were the first self-evaluation reports to be submitted in the overall accreditation process, which was set in motion just this year (2014). Sufficient evidence was provided in relation to the large majority of the standards and standard sections in the nine self-evaluation reports that were submitted in order for a formal QA procedure to be carried out, and there were no cases where a lack of evidence postponed or delayed a site visit.
The experience over this initial period of several months will, at all events, serve to identify and incorporate the necessary improvements and enhancements to make accreditation more effective and efficient, to the benefit of all of the stakeholders involved.