A bimonthly publication of AQU Catalunya

43

July 2009

One of the elements underlying the present-day university culture in Spain and Catalonia is the belief that the activity, repercussions and quality of research work is measured in a highly objective way and very precisely by the bibliometric indices that E. Garfield, director of the Institute for Scientific Information (ISI), published during the eighties in the Current Contents publication.

Many researchers and university administrators are convinced of this, on the basis of the assumptions that the sole mission of research is to contribute to the advance of universal scientific knowledge through the publication of results in English, and that peer review provides an almost total guarantee of quality in publications.

As a consequence of this belief, the management of research (budget allocations, professional career, categorising of research institutions, groups and researchers, etc.) based on these indicators is now unquestioned (to the detriment of other results of scientific activity, such as patents, transfer to the production sector, etc.). Much has been written on this phenomenon, which is introduced here merely as an example of the enormous effects of the indicators used to measure quality on the outcomes of public investment and personal effort.

A second element of the present-day university culture, which has been diffused just as much as the one above, is that of the difficulty, if not impossibility, of measuring the outcomes and quality of teaching.

This belief is based on facts as real as the fragmentation into subjects of the assessment of learning processes, which impedes the assessment of benchmark-based learning, and poor definition of the competence profile of graduates. The insufficient evaluation of early professional development and employment and labour market outcomes as a means to indirectly measure this does little to help in this regard. It must be recognised that some of these realities may no longer apply with implementation of many of the principles of the European Higher Education Area.

The coinciding of these two beliefs has contributed to the progressive devaluation of the teaching competence of academic staff, together with their loss their interest in enhancement and the updating of teaching methodologies.

The introduction by AQU Catalunya of criteria for indirect assessment of the quality of individual teaching competence and the subsequent development of indicator catalogues for objective measurement using teaching competence evaluation handbooks for Catalan universities is an important counterpoint to this belief.

The application of these criteria has so far not led to any great cultural change in how teaching by university academic staff is considered or assessed. Criticism, on the basis of absolute value considerations, of the processes linked to administration is very high in university academic staff. It must be recognised, however, that there is a considerable level of acceptance regarding assessment processes that are being carried out, in spite of the fact that the indicators employed usually generate discussion, with even the criteria frequently being subject to intense debate. Nevertheless, analysis of the ongoing trend in the comparable results over various assessments does show that changes are taking place in the behaviour of academic staff with regard to adaptation to assessment criteria.

This particular procedure has been used at the University of Lleida (as in other Catalan universities) from 2003 onwards. The handbook, indicators and software procedures have been adapted every year since the first assessment up until the present day, which can be seen as constant enhancement that is grouped into five main areas: the quality of the basic information sources for the indicators; clarification and objectivity in the objective information that the participants in assessments are asked to provide (deans, heads of department, Institute of Educational Sciences¹), adequacy of the distribution of the results obtained with the different indicators, ease of use of the software used by academic staff, and improvement of the information on the process.

We have also recently been contributing to the availability of enhancement resources for academic staff that have with problems with the indicators. These changes are all based on innovations in the procedure provided by AQU Catalunya (e.g. the concentration of criteria), in the assessment received as a result of the accreditation procedure, and above all through the analysis made on a year-on-year basis of the results of the procedure.

Improvement of the procedure made it possible in 2008 to apply a criterion of excellence as the basis for an individual award for quality in teaching. The university's Social Council had been suggesting this action for some time, although it was postponed due to the lack of general conviction in the preciseness of the evaluation method. The award consists of an honorary "mention" in the form of a diploma, which is publicly presented to academic staff awarded the recognition in a ceremony at the beginning of the academic year.

The criterion involves the university's Assessment Committee coming to agreement on the overall merits of the teacher who gains the highest results in all of the criteria set out in the Teaching Competence Evaluation Handbook. The two times that this has been held, the Assessment Committee came to a positive agreement regarding all of the teachers (2 and 3 respectively) who obtained these results, which we interpret as a criterion of the procedure's sensitivity and specificity in assessing the concept of quality in teaching. Other indicators supporting the close alignment of the assessment with the concept are the largely positive opinion of the university community concerning the merits of teachers being awarded the mention, and the standard distribution of the final results, which shows how discriminating the procedure is.

It is still early days to make an assessment of the real effects of this honorary mention, although its good acceptance from the outset should undoubtedly be seen as agreement regarding the adequacy of the general procedure in measuring quality in teaching by members of the university community. It remains to be seen what motivating effect this has on academic staff who are particularly interested in demonstrating their excellence as teachers, together with the consequences and possible applications of the existence of a clearly delimited population of teachers who have received public recognition (a mention) of the highest quality. The aspiration of assessment with clear and accepted criteria for quality in teaching that has consequences for teaching similar to those that bibliometric indices have had on research is possibly becoming slightly more justifiable.