A bimonthly publication of AQU Catalunya

75

November 2014

The commitment of AQU Catalunya to improving the effectiveness and efficiency of quality assurance takes the form of regular meta-evaluation actions, which act as feedback and deliberative mechanisms for QA development and improvement.

In the context of the accreditation of recognised First and Second Cycle (undergraduate and Master's level) university degree programmes, on 17 September AQU Catalunya organised a meta-evaluation session, based on the experience gained from the nine external site visits carried out so far in relation to programme accreditation between April-July 2014. Stakeholders in accreditation were widely represented among the approximately fifty participants in the session and included programme coordinators, heads of QA units and offices, and members of the external review panels.

The proceedings of the session were organised according to:

• The accreditation guidebook.
• Producing the self-evaluation report and selection of the associated evidence.
• Carrying out of the actual external site visit.
• Findings and outcomes of the site visit.

An analysis is given below of the quality of the evidence provided by the faculties and schools in the self-evaluation reports, a key factor in ensuring the quality of the accreditation process in general and that has to do with the aspect dealt with in point two of the session on 17 September.

The purpose of this analysis is to highlight the main areas for improvement identified by the external review panels. For the analysis, use was made of the review reports on the self-assessment report and evidence, which are the reports drawn up by the external review panels prior to the site visit and where the faculty or school is requested, where applicable, to provide new documentation or additional information.

The requests by the external review panels for further or additional information in relation to self-evaluation reports submitted during the first half of 2014 were as follows:

1. Quality of the study programme

With regard to this first standard, a request for information on, and an assessment of, the profile of new entry students to be included in the self-evaluation report was made on six occasions. This information can be summarised as:

• Aspects concerning the adequacy of the entry requirements for new students and the coherence between the number of students admitted and the number of places offered.
• The satisfaction of teaching staff regarding the relation between stated and actual learning outcomes.

To a lesser degree (two occasions) information was requested regarding teaching coordination mechanisms.

2. Relevance of the public information

In relation to the second standard, in the majority of the applications (seven) there were shortcomings detected in the information provided on the website of either the institution or the study programme.

Requests were made for:

• Up-to-date information on the characteristics and delivery of the study programme.
• Course guides.
• Plan for interactive tutorials.
• Indicators.
• Study programme monitoring progress reports.

On one occasion documentation on the procedures in the internal quality assurance system dealing with public information, the gathering of information and accountability was also requested.

3. Efficacy of the study programme's internal quality assurance system 

The third standard, referring to the internal quality assurance system and its efficacy, resulted in a large number of requests for the information provided to be either supplemented or modified.
As for the instruments for gathering information regarding stakeholder satisfaction, eight requests were made for the following to be added to the self-evaluation report:

• Examples of the questionnaires dealing with the satisfaction of both the students and teaching and research staff.
• Survey response rates and survey frequency.

The request for improvements to be made to the quantitative information on programme delivery was also made on eight occasions, in the form of:

• Time series data.
• Improvements in the definition of the indicators.
• The disaggregation of data (for example, the number of full and part-time students and teaching staff).
• The corresponding indicators for the analysis and assessment to be submitted.

With regard to the review and revision of the internal quality assurance system, detailed information on the implementation of procedures and the time limits/deadlines for implementation was requested on four occasions.

Lastly, the plans for, and monitoring of, improvements and corrective action were commented by the external review panels on nine occasions. The main requests were:

• The break down of actions that are generic into specific actions.
• To incorporate indicators on the fulfilment of actions with a specific timetable.
• The coding of actions and defining of specific individuals who are responsible for them.
• Improvements to traceability as regards previous action plans.
• Improvements to the coherence between the plan for improvements and the line of reasoning given in the self-evaluation reports.

4. The suitability (qualifications and level of competence) of teaching staff on the study programme

Regarding standard 4, in reference to academic staff, the external review panels in particular called for more extensive qualitative and quantitative information on the experience of teaching and research staff in teaching, research and at professional level.

Secondly, seven requests were made in connection with the suitability and level of competence of academic staff on study programmes, with:

• Summary charts on teaching staff.
• A breakdown of the information on teaching and research staff assigned to first year studies, practicals and the First and Second Cycle final year project/dissertation.
• Specifications of the part/full-time assignment of teaching and research staff on the study programme.
• Comments and feedback on the quality and proficiency of the current teaching staff.

Requests were also made for new information regarding the matter of the institutional support received by academic staff, more specifically:

• Staff training courses undertaken, an assessment of the courses, and the level of satisfaction with the courses.
• Data on staff participation in training courses.

5. Efficacy of learning support systems

Relatively speaking, standard 5 is the one where fewer requests were made for more extensive documentation. Requests were made on three occasions for an analysis and comments on the workings of the plan for interactive tutorials to be included in the self-evaluation report.

To a lesser degree, the external review panels requested indicators on the level of satisfaction with academic tutorials, satisfaction indicators on careers and professional guidance, and documentation from the internal quality assurance system on student support and guidance.

6. Quality of programme learning outcomes

Requests for new information were made in relation to two sections of standard 6. The highest number (seven occasions) of requests were for examples of the students' achievements, more specifically:

• Provide the selection criteria for the courses taken and the examples/samples submitted.
• Submit the reports on compulsory internships/job placement.
• Provide the list of Second Cycle (Master's) final year dissertations submitted the previous year, with information on the type, subject and qualifications.

Requests in the second section were for the trends, together with an evaluative analysis, in the indicators of both the performance of graduates of the study programme and their transition to the labour market.

Conclusion

The overall assessment by the external review panels of the quality of the self-evaluation reports and other supporting evidence connected with the site visits is, in general, quite positive, bearing in mind in particular that these were the first self-evaluation reports to be submitted in the overall accreditation process, which was set in motion just this year (2014). Sufficient evidence was provided in relation to the large majority of the standards and standard sections in the nine self-evaluation reports that were submitted in order for a formal QA procedure to be carried out, and there were no cases where a lack of evidence postponed or delayed a site visit.

The experience over this initial period of several months will, at all events, serve to identify and incorporate the necessary improvements and enhancements to make accreditation more effective and efficient, to the benefit of all of the stakeholders involved.