HUMANE
Heads
of University Management & Administration Network in Europe
SEMINAR
Ruprecht-Karls-Universität
Heidelberg
Friday
25th to Saturday 26th October 2002
The presentation will first describe the
objectives and functions of the Scientific Commission – notably that of developing
the structure of teaching and research in the Higher Education system of Lower
Saxony. This includes developing and
assessing centres of excellence in research, and also developing forms of
research assessment. The Commission
also provides advice on the allocation of state funding
The system of Research Assessment
(RAE) will be outlined. The RAE is
conducted by subject area, not by institution, and is based on informed peer
review. The assessment is conducted
independently of universities and of political involvement, and there is no
direct link between RAE results and funding.
The methods rely on initial self-reporting by departments, with the
nomination of peer-groups and a process of audit. Universities can make comments before the reports are released by
the Research Council.
The criteria which are employed include the
quality and relevance of research in the particular subject area within the
university (both in national and international terms), and the quality of that
research relative to financial input.
On the basis of such analyses, ratings are given.
Since the introduction of the scheme in 1999
several lessons have been learned. For
example, the reputation of the peer assessors is a crucial factor for
acceptance of the system. Quality assurance must be based on scientific, not
administrative, political or other standards, but the RAE must take
subject-related standards into account.
Both Ministry and university leaders must be prepared to respond to
recommendations.
Future work in this area will recognise that
the RAE requires a lot of time and effort from universities and academic peers,
but that there is a growing demand for reliable information as the basis for
stable planning and funding. There must
be a balance between standardisation and the appropriateness of RAE procedures
to specific disciplines, and there is a need to monitor both the intended and
the unintended effects of the exercise.
The Experience of
the University of Osnabrück with the Scientific Commission Evaluation
Christoph Ehrenberg, University of
Osnabrück
When the Academic Commission of Lower Saxony (WKN) was
founded five years ago, it was, like all new institutions, met not only with
positive expectations but also with scepticism and concern. For example:
·
What would the consequences of the Commission’s work be for research
activities at institutions of higher education?
·
Would it lead to promotions or cuts? Or a mixture of both within the
framework of a structural research policy developed by the Land?
·
Or would the results of the evaluation be without consequence, as has
been the case with previous research reports and structural reports by experts
commissioned by the Land, only to be
put on the shelf?
·
Would the work and energy put into the evaluation by the
institutions of higher education (writing of self-assessment reports, delivery
of data, co-operation in on-site evaluation) be proportionate to the results
gained?
·
How would the research evaluation carried out by the WKN stand with
regard to other evaluations, in particular in the field of teaching, carried
out by the Central Evaluation Agency in Hanover?
Research achievements traditionally play an important
role at German universities. Research activities and their expert evaluation
are crucial to academic careers, in particular in the nomination of professors.
The same goes for the acquisition of third-party (external) funding. Such
third-party funding is only granted by large funding institutions,
predominantly the German Research Foundation (DFG), on the basis of intensive
evaluation of previous research achievements and planned projects. However, it
must be pointed out that there is a clear distinction between the specialist
disciplines: while the natural sciences are fundamentally dependent on
third-party funding, academics in the humanities are still predominantly of the
Humboldt type, undertaking research in freedom and loneliness. As such, for most academics in the
humanities research evaluation was a step into a completely new world.
Today, after five years, an overall positive balance
can be drawn for research evaluation in Lower Saxony. This applies on the one
hand and predominantly to the work of the WKN. The research evaluation is
viewed as an important element of quality control and development planning in
the higher education institutions. Negative aspects still remain, however, in
the implementation of the evaluation results by the Land, in particular with regard to the structure of the university
and research system in Lower Saxony.
Session 2
Evaluation
in Union - The Joint Evaluation of Educational Programmes
Since
the late 1980‘s and beginning of the 90‘s „Evaluation“ has become more and more
important at German universities. Gradually German universities have taken over
more autonomy and self-responsibility in research, study programmes and
administration instead of relying on governmental directives. This development
has forced the necessity of creating a system of quality assurance - that is, Evaluation
(looking at the process) and Accreditation (looking at the results).
In
February 1998 the presidents and rectors of the Technische Universität
Darmstadt, the Universität Kaiserslautern, the Universität Karlsruhe and the
Eidgenössische Technische Hochschule Zürich agreed to a joint evaluation
project whereby the three German universities would undergo comparative
evaluation processes within their disciplines under the moderation of ETH Zürich
at every step of the procedure.
The
relationship between input and output has been very positive. The procedure
combines internal and external components, optimises the mutual learning
process and is economical because of its self-organisation. Finally there is a
high level of identification with the procedure and results within the
departments.
Session 3
External quality assurance of
Higher Education in the UK became a central feature of the funding model under
legislation introduced in 1992. Over the last 10 years, this has been a
controversial issue which has led to concerns about the accuracy and value of
published quality data; the administrative burden imposed by external audit;
the funding implications of quality; and the relationship between government,
funding bodies and institutions.
Scotland is about to embark on a
new approach to quality in Higher Education, developed by the Scottish HE
Funding Council (SHEFC). The approach is informed by a fundamental analysis of
quality assurance and enhancement, and of the different roles of institutions,
students, and national policy bodies in quality processes. Our new approach is
still being developed and will be fully introduced in 2003.
The Scottish approach is based on
a number of key premises:
· All
parties should share a common vision of what we mean by ‘a high-quality higher
education system’
· Quality
must be owned by the institutions which deliver it, and not by funding or
regulatory bodies
· Quality
systems must take account of the needs of students, and provide opportunities
for the student voice to be heard at all levels
· External
quality assurance should be universal in principle but selective in practice,
based on a balance of risk and cost
· Processes
which are based on quality assurance are not ideal for promoting quality
enhancement
The paper will discuss the
implications of each of these premises.
Our experience has identified a
range of important issues which may be of interest to this audience, and these
may provide useful points for discussion at the seminar. These include:
· the
status of students within universities (are they customers or not?);
· the concept
of ‘reflective practitioners’ as a model for continuous quality enhancement;
· the
balance between ‘teaching’ and ‘learning’ activities within institutions;
· the
role of external viewpoints in institutional quality systems; and
· the
scope for institution-wide approaches to quality within highly-devolved
academic structures
Funding
Council Evaluations – the Experience at Aberystwyth
Noel
Lloyd, University of Wales Aberystwyth
I shall describe some of the experiences of Aberystwyth in relation to Funding Council assessments of Teaching and Research. I should point out that there are some differences between the approaches adopted by the English, Welsh and Scottish Funding Councils.
(i)
The programme
of direct evaluation of individual subject areas (‘Teaching Quality
Assessments’ or TQA) concentrated on process. A great deal of effort was required to prepare for these events;
we found wide variation in the value of these exercises, and much depended on
the individual members of the
panels. The system of
institutional observers introduced in
Wales was particularly helpful. A
distinction is drawn between ‘quality’ and ‘standards’.
(ii)
In parallel
with TQA there has been a programme of Institutional Audit. Again much depended on the individual
members of the panels. I shall comment
on the methodology adopted.
(iii)
Research
Assessment : this has had a profound influence on academic planning in
universities which see themselves as ‘research-led’ in some sense, and has led
to quite an intrusive level of research monitoring. I have no doubt that an improvement in the overall quality of
research has occurred, but the future pattern is now being considered.
(iv)
I shall also
comment on the increasing tendency of hypothecated funding (‘something for
something’) and the intrusive nature of the Funding Councils’ monitoring of
programmes which are funded in this way.
Quality Evaluation in England – The
Future
Jane Hopkinson, University of
Plymouth
The
system of teaching evaluation in Wales (discussed in the previous presentation)
has closely mirrored that operating in England. Following considerable concern within institutions at the burden
placed on teaching staff and institutional resources as a result of those
particular forms of review, a new system has been developed and is to be
introduced from the current academic session, 2002-03.
The
two bodies most obviously involved in this process are the Higher Education
Funding Council for England (HEFCE) and the Quality Assurance Agency (QAA).
In
summary, the new system will involve an ‘institutional audit ‘, conducted by
QAA, and, in the initial transition period (2002 – 2005) ‘developmental
engagements’ focused on specific subject areas. The developmental
engagement is similar to the old-style subject review, in that it
involves sending QAA a short self-evaluation document prepared by the subject
team and a subsequent visit by a team of reviewers. However, the key differences are that:
· the University Students’ Union may also
make a separate and potentially confidential submission.
· the University may nominate a fourth
member to the review team, from its own staff
· the review team will communicate
largely by email or telephone rather than meeting, except for the two days of
visits to the University.
· the outcome will be based on
‘threshold’ judgements of academic standards and learning opportunities, not on
numeric scores
Institutional audit is similar to ‘old-style’ continuation audit except
that:
· the Students’ Union may again make a
separate and confidential submission
· the audit will also involve a number of
discipline audit trails, for which the subject areas will have to prepare self
evaluation documents on the part of individuals.
At
the same time, there will be an increased focus on ‘customer evaluation’. A HEFCE Report: Information on Quality and Standards in Higher Education requires
institutions to publish on their Web site detailed information about their
courses. This includes quantitative
information such as statistical data on the entry qualifications and degree
classifications of students,[1]
and benchmark comparators on things like progression and completion rates,[2]
or first destinations (that is, graduate employment statistics)[3]. It also includes qualitative
information such as reports from external examiners’ reports, summaries of
institutional quality review processes, student feedback, and learning and
teaching strategies. This is intended
to enable potential and current students to make informed choices, and is a
major part of Government policy in this area.
The
presentation will detail some of these changes and explore implications for
institutions.
Note:
Full details of the QAA process can be found on the QAA website:
www.qaa.ac.uk (entitled Handbook for Institutional
Audit)
Full details of the HEFCE
publication can be found on the HEFCE website: www.hefce.ac.uk
Session 4
The wave of Evaluation and Quality-Techniques that
began invading European universities in the 1990’s has met with very different
responses in our universities. The patterns of Evaluation methods though, have
been similar in many countries.
Evaluation systems, like Quality Plans, are tools or
techniques that can be used in order to focus an organisation, and to involve
its workers. They can also completely change the way of working.
The success of this Evaluation Process, like any other
road to change, needs considerable technical support which includes the
adaptation of information systems and an alliance with university
services. But it also demands constant
leadership in order to maintain the organisational effort that is necessary in
order to obtain the best results.
The role of the Head of Administration can and should
be very important as a leader, at the very heart of this journey towards change
based on the philosophy of quality.
Even though the goals of the Bologna Conference still
await us, Quality and Evaluation Plans remain a challenge for Heads of
Administration, and a lever that could be very effective if used correctly, to
help our universities adapt to the demands of a new century.
Session
5
The Evolution and Assessment of the ELT
(English Language Teaching) Service Systems at the
University of Pavia.
Reforms introduced in the last two or three years in the
legislation governing Italian universities require all students to be assessed
in terms of their skills in a European foreign language. From being a small
centre concerned with a few specialist degree courses, the Language Centre,
like all University Language Centres in Italy, was called on to scale up its
services to a point where it could provide a University-wide English language
teaching and testing service.
The
presentation describes how, in the space of three months and using existing
resources, a new teaching and testing system was built, and how the results
from research into computer-based teaching were integrated into the service
delivery system. In particular, the system focused on creating a standard
product that could be reproduced throughout the University and which could also
be delivered via web services to students' homes. The system uses streaming
video and relational database technology.
The
first part of the presentation will illustrate the system and the innovative
technology using a visual, as well as a linguistic, approach. The second part
will be concerned with evaluating the new system and understanding its further
capacity for growth. What new staffing, and what financial, organisational,
research and other resources will be needed? What further development of the
technology? What training of staff? And above all, how can an overall
evaluation strategy be developed?
The
presentation will describe efforts, still at a skeletal stage, to build an
evaluation system capable of measuring the system's success and its overall
contribution to the goals of University education. But how well can these
evaluation strategies measure the system's future potential - its capacity to
grow and to incorporate, for example, the results of research? These matters
will hopefully be the subject of general discussion.
[1] The final degree obtained by students in the UK has traditionally been divided into four main classes (a First, Upper or Lower Second, or Third class degree. In the past these disstinctions were of use mainly to the student. Now – and especially when correlations are made between such outputs and the inputs of student qualifications on entry, they are of significant interest also to the institution.
[2] Progression rates will measure the successful movement from one year (or module) of a course to the next; the completion rate will show the proportion of students who complete the nominated degree.
[3] Official surveys generally take the student’s „occupation“ six months after graduation as the „first destination“.