GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
Ref. Ares(2016)3996406 - 29/07/2016
ERASMUS+
PRACTICAL GUIDELINES FOR NATIONAL AGENCIES
ON
QUALITY ASSESSMENT OF ERASMUS+ ACTIONS
Version 23/04/2015
1
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
1. Purpose
The purpose of this document is to provide the National Agency (NA) with practical guidelines on
the management of the quality assessment of applications for Erasmus+ decentralised actions.
These guidelines complement the minimum requirements for grant award set out in the Guide for
NAs. Their aim is to provide support to the NA in order to manage the quality assessment in a
highly qualitative and comparable manner across NAs.
These guidelines are complemented as well by the Guide for Experts, which provides experts
undertaking quality assessment for the National Agency with the necessary guidelines and
instructions to deliver the expected high quality work. The Guide for Experts is publicised on the
website of the Commission in complement to the Programme Guide, in order to provide both
experts and applicants with the same and transparent information on the quality assessment that
applications will undergo.
In case of differences between the above documents, the Programme Guide prevails over the
Guide for Experts, which prevails over the Guide for NAs, which prevails over the present practical
guidelines on quality assessment.
2. Recruitment of experts
As the Guide for NAs explains, the NA is required to call upon independent experts to undertake
the quality assessment of applications. Depending on the Programme action and level of grant
support requested, the NA must involve at least one or two experts, whereby in some cases
experts external to the NA must be used.
In case of external experts, the NA has to see to it that it attracts persons with the right profile
considering the action(s) for which the experts will undertake the quality assessment. The NA,
thus, has to duly consider the fields of education, training and youth, in which the experts must
have the necessary expertise to be able to judge award criteria such as relevance and impact in
relation to both the objectives of the action and the policy priorities that apply.
The NA may work with external experts who are already experienced in quality assessment of
applications in the predecessor programmes. However, the NA should renew the pool of external
experts regularly, in order to complement experience with fresh views.
The NA can organise the recruitment of experts in different ways depending on the national
context. However, in order to guarantee a fair and transparent quality assessment, it is
recommended to work with open calls for external experts, based on objective requirements in
terms of expert profile sought. Thus, the requirements should specify at least the expected
professional background, experience, foreign language skills, experience in assessing international
cooperation projects, but also elements such as experience in international project cooperation or
in policy priority areas may be included.
N.B.
The same quality requirements apply to NA staff involved in the quality assessment.
Depending on the size of the country and the pool of organisations from which experts may be
recruited, the NA could also call on experts from other Programme Countries (in particular for
Strategic Partnerships), either directly, or by sharing its pool of experts with NAs in other countries.
2
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
Considering the selection calendar and expected number of applications, the NA should make sure
to establish a sufficiently large pool of external experts in order to be able to finalise the quality
assessment stage timely.
3. Appointment of experts
In the case quality assessment is undertaken by NA staff members, no specific appointment is
required. However, the NA management has to ensure that the required segregation of duties is
respected and the NA staff concerned will be required to sign a declaration on the absence of
conflict of interest (see details in section 6 below).
As regards the recruitment of external experts, the conditions under which the quality assessment
will have to be undertaken will have to be clear for the NA and the experts from the outset. Thus,
the NA has to formalise their appointment. For that, it may use different forms depending on the
national legislation, e.g. an appointment letter or contract signed by both parties. When an open
call is used to recruit experts, the terms of engagement should be published upfront, so that only
persons ready to accept those conditions apply to the call.
The minimum elements that the appointment letter or contract should comprise are the following:
•
prevention of conflicts of interest
•
obligation of confidentiality
•
tasks of the expert
•
time for assessment
•
remuneration arrangements in case of paid experts
•
quality requirements for assessing applications
•
practical requirements such as internet access for remote assessment
The appointment letter or contract should be established/signed before the experts start to work.
4. Briefing of experts
Before experts start to assess applications, the NA has to instruct them clearly on the tasks to be
undertaken and provide them with all the relevant background documents.
As set out in the Guide for Experts, they have to become familiar with all the Programme reference
documents, in particular the Programme Guide, the application forms, the Guide for Experts, the
assessment forms and the OEET User Guide (the Online Expert Evaluation Tool, see below).
If relevant, the experts should refer in particular to:
the “Practical Guidelines for the assessment procedure for the mobility between
programme and partner countries in the higher education field” to insure that experts
understand the specificities of this type of mobility
3
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
the guidelines for the selection of KA2 school-to-school Strategic Partnerships, to have an
appreciation of the specificities of this type of action.
The NA can organise the briefing and training in different formats, from onsite meetings bringing
together all experts to bilateral sessions, online briefings, video-tutorials or a combination of
these. However, for new experts, it is strongly recommended that live meetings are organised in
order to provide the necessary time and space for productive interaction so that experts are
adequately prepared for their task. Group training is also preferable over individual training in
order to favour peer-learning between experts. It may be also useful to ask former experts with
whom the NA had a good experience to provide support for the training of new experts.
Given the importance of the relevance and impact criteria in Erasmus+, the NA should make sure
that the training stresses these criteria accordingly. In this context, it is important that the NA is
able to well explain the applicable action objectives and policy priorities to experts and it should
therefore familiarise itself with those in detail.
In order to reach comparable quality levels of assessment, the training should comprise an
element of practical exercise, having for example all experts assess the same application, which
allows for a detailed discussion of the assessment results with the NA and among experts in order
to clarify which elements to consider when assessing the applicable award criteria. In this context
it is also particularly important to draw the experts' attention to the proportionality principle that
has to be applied, in order to assess grant applications on their own merit and with due regard to
the organisations participating in the project, considering their experience and capabilities.
As the quality assessment will be done using the OEET (Online Expert Evaluation Tool), the NA also
has to integrate an element of technical training of experts. The NA can make use also of parts of
the video-tutorials that the Commission will make available on the use of OEET.
The Guide for Experts sets out how experts are expected to assess the award criteria in terms of
scoring and providing comments on each award criterion. During the briefing, experts should not
only be made aware that they have to provide clear, consistent and balanced comments on each
award criterion, but also that they need to do that in a polite and constructive manner, so that the
NA can use these comments directly for feedback to the applicants.
In order to reach a genuine quality spread of applications, experts should aim to score criteria as
accurately as possible, and avoid giving average scores across the board.
It should also be pointed out to experts that as part of their quality assessment they are asked to
analyse the grant requested and suggest any reduction if necessary (especially in the case of KA107
where mobility flows can be ineligible or disproportionate). However, experts should also be
informed that past experience shows that serious grant cuts often result in difficulties for projects
to realise their objectives. Therefore, experts should be advised to consider carefully whether to
suggest a grant reduction or refrain from it and only score the proposal lower on the relevant
award criterion in case of incoherence between the grant request on the one hand and the
objectives, activities and outputs/outcomes proposed by the project. Experts should thoroughly
substantiate any proposed grant reduction and quantify the proposed reduction for the specific
grant items. Given that grants are largely unit cost based, the focus should be on the units
proposed rather than on the amount of grant request per se (e.g. number of mobilities for
transnational project meetings, number of days per staff category for intellectual outputs, etc.).
The expert will have to explain the incoherence noted in the comment on the award criterion, but
detail the proposed grant reduction in the comments addressed exclusively to the NA.
4
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
In cases where applications have to be assessed by two experts, the assessment results will have to
be consolidated in order to provide a unified single feedback to applicants on any application.
Experts should be made aware that they will be required to cooperate in pairs in order to reach
agreement for this consolidated assessment.
Experts should also be made aware that they are expected to work to high quality standards. This
means that the NA will monitor their work and may ask them to redo/improve an assessment if
the result is not of sufficient quality and that they may not be paid for work of insufficient quality.
The NA should clearly instruct experts what to do in case they discover during the assessment
process that they have a potential or real conflict of interest in relation to a specific application
submitted for the selection round under assessment.
Experts should also be notified clearly that they are not allowed to contact the applicant directly if
they come across a clerical error in an application for which further information or clarification
may be requested from applicants. Experts should know that such cases should be referred to the
NA, which will decide on the necessary follow-up. Any contacts with applicants can be made
exclusively by the NA and experts not respecting this principle should be excluded from the
assessment process.
The NA should draw the experts' attention to other elements of assessment that they will have to
provide during the quality assessment. This refers to content elements on which information is
collected from experts for statistical purposes. These content elements are included in OEET and
will be marked by the experts directly in the tool when doing the quality assessment.
Finally, experts should be informed that in addition to their comments on the scores and the
application as a whole, they can give direct feedback in OEET also to the NA in case they want to
raise any specific issue of concern or doubt in relation to the application under assessment. The
comments to the NA are clearly distinct from those aimed at the applicant.
5. Organisation of the work of experts
As the quality assessment will be done on line using OEET, the NA can have external experts assess
applications remotely. Experts should be made well aware of the confidentiality requirements and
their responsibility to prevent unauthorised access to the OEET or to any data downloaded from
the tool.
If the NA prefers experts to assess applications in the premises of the NA, it has to foresee the
necessary access to computers so that experts can work directly in OEET.
When distributing applications for assessment to experts, the NA has to take into account their
professional profile, language skills, experience in assessing international projects etc. to match
experts as well as possible to the type of applications to be assessed. Where more than two
experts are needed, the NA is recommended to establish varied "pairs of experts", mixing
experienced with less experienced ones, and alternating persons in pairs, so that an as broad mix
as possible is reached in order to ensure coherent and balanced quality assessments.
5
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
6. Prevention of conflicts of interest
As the Guide for NAs stipulates, the quality assessment must be fair and impartial. This implies that
persons with a potential conflict of interest cannot take part in the quality assessment.
In formal terms, each expert will have to sign a declaration of honour regarding the absence of
conflicts of interest and the obligation of confidentiality. Similarly, the quality assessment form in
OEET will comprise a declaration on the absence of conflict of interest on the particular application
under assessment. The meaning of a conflict of interest is defined in the Financial Regulation and
may arise basically from any kind of connexion an expert (whether internal or external to the NA)
may have with any organisation or person participating in a given selection round or specific
application.
In practice, a number of cases may occur where the NA will have to make a judgment based on all
information elements that it has at its disposal to decide whether the expert should (dis)continue
the assessment of a specific application or (dis)continue the assessment of any application under
the given selection round.
Experts have to be excluded from all assessment for a given selection round if they themselves are
directly involved in any of the applications submitted. The same normally applies if the
organisation by which they are employed is directly involved in any of the applications submitted;
however, in this case the specific circumstances can be considered by the NA. For example in the
case of a large university, the NA may decide to allow an expert employed by that university to
continue assessing other grant applications under the selection round if there is no direct link
between the expert and the department or faculty involved in an application submitted under that
round.
At any rate, the NA will have to be very careful to prevent any potential conflict of interest and
establish the necessary safeguards taking into consideration the environment in which it is
operating.
In the case of countries with a coordinated management between different NAs under a single
Delegation Agreement, the NA has to ensure that no conflict of interest exists for experts across
the different NAs. This relates in particular to actions for which applications can be submitted to
different NAs (cf. cross-sectoral Strategic Partnerships).
7. Consolidation of assessment results
Before the assessment results can be proposed to the Evaluation Committee, a consolidation
needs to take place.
In case an application is assessed by only one expert – whether internal or external to the NA – no
consolidation is required and the results of that single assessment determine the final score and
assessment comments.
However, in case an application is assessed by two experts – whether internal or external to the NA
– the results of those two assessments must be consolidated into a single final score and
assessment comments (both comments for individual criteria and overall comments). In such case,
the NA designates one of the two experts to consolidate both assessments. This expert will
become the so-called "lead expert".
Depending on the degree of discrepancy between both experts' assessments, different
6
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
consolidation rules will apply:
limited discrepancy: a difference of less than 30 points in the overall score;
significant discrepancy: a difference of minimum 30 points except if both assessments show
that the proposal does not pass the thresholds for acceptance.
In case of
limited discrepancy, the lead expert does the consolidation of the score and the
assessment comments in agreement with the other expert, of which proof should be provided to
the NA. In case the two experts fail to agree on the consolidation, the lead expert should inform
the NA accordingly. The NA will then have to decide whether it is worthwhile pursuing a
consolidation between the two experts or to provide the application for an independent
assessment to a third expert; in the latter case, the approach for significant discrepancy applies as
set out hereafter.
In case of a
significant discrepancy, the NA must designate a third expert for an independent
additional assessment of the application. The consolidation will be based in that case on the
assessment of the third expert and the one of the two initial ones whose score is closest to that of
the third assessment. The NA designates either the third expert or the initial expert whose
assessment is closest to the third assessment as lead expert, who will undertake the consolidation
in agreement with the other expert.
Nota bene
Consolidation must be undertaken by the experts having done the initial assessment and thus
knowing the application in detail. However, in exceptional justified cases, the NA may undertake
the consolidation directly without involvement of the external experts having assessed the
application. In such case the NA staff member undertaking the consolidation has to be designated
as third expert in OEET and be given the role of "lead expert" in the tool. However, as OEET allows
for maximum 3 experts per application, this option does not apply when an application has already
been assessed by 3 experts. All cases of consolidation having been carried out by a NA staff
member who has not been appointed expert for the application in question must be duly recorded
for the sake of audit trail.
OEET will be configured in the following standard manner for the consolidation. The tool will:
calculate the average of both experts' scores
show the assessment comments of both experts
The lead expert will see this information on the screen but will be able to change both the scores
of the individual award criteria and the assessment comments during the consolidation. The
mathematical average is only provided as a basis for the lead expert to arrive at the final score but
he/she should use his/her own judgement, taking into account the scores and comments given in
both individual assessments and in agreement with the other expert.
The NA can advise experts to use decimals or half points in their consolidated assessment to
reduce the number of several applications with the same score.
The final total score will be recalculated automatically based on the modified scores of individual
award criteria.
As regards the information for statistical data collection on which the opinion of the expert is
requested during the quality assessment, experts should agree on the final answer if they provided
7
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
different answers in their individual assessments. In case of disagreement of both experts,
the NA may either re-assess the issue itself or ask another expert to re-assess the application as
third expert.
8. Monitoring and feedback to experts
In OEET, the NA will be able to monitor the progress of experts in the quality assessment (see OEET
User Guide for the technical details). Given the tight deadlines for the whole selection process, the
NA should monitor the progress closely in order to intervene timely in case of delays in assessment
by any of the experts. This is particularly important in relation to the selection of Strategic
Partnerships, where NAs need to finalise the quality assessment at the same time in order to be
able to undertake the validation of participating organisations in projects proposed for funding in
due time before the grant award decision can be taken. Considering these tight deadlines, the NA
has to make sure it plans well also the timing of the Evaluation Committee consultations before the
grant award decision can be taken.
In order to reach good quality assessment results, it is important that the NA monitors the results
of the experts' work. Note that the NA staff member managing and supervising the assessment
process (holding the role of Selection Manager in OEET) for a given selection round must not be
involved in assessing applications under this round.
This means that assessment comments and scores are thoroughly checked for quality and
coherence once the assessment is submitted in OEET and that the NA provides direct feedback to
the expert in case of problems, in order to make sure that the outputs of assessment reach the
desired quality level.
As indicated above, the quality condition should be integrated as well in the appointment letter or
contract that the NA concludes with any external expert, so that this element can be used in case
of difficulties encountered.
If necessary, the NA may ask an expert to redo his/her assessment. OEET provides for the
possibility to re-open assessments already submitted by the expert before the consolidation task
starts. Similarly, the consolidated assessment, once submitted, can be re-opened if necessary.
The NA should also put in place a helpdesk for the experts in case they have any questions on the
content or practical/technical arrangements for the quality assessment.
It is good practice to provide a collective and/or personal feedback to experts after a given
selection round with the lessons learnt from the exercise. This feedback can be used in turn for the
training of experts in future.
9. Feedback to applicants
The NA must give feedback to all applicants (accepted, rejected and on the reserve list).
For applications
that did not meet the minimum quality criteria;
that are not proposed for funding; or
for which the NA proposes a reduced the budget (compared to the one requested by the
applicant) due to the qualitative evaluation
8
GfNA-III.2 – Erasmus+ Guidelines for NAs on quality assessment – version 23 April 2015
the NA shall include the overall score and comments on the application, as well as scores and
comments on individual award criteria.
The NA staff will be able to edit the consolidated comments given by experts in the appropriate
wording and style as necessary before giving feedback to applicants. It should, however, take care
not to modify the sense of the comments. The NA shall not change the scores.
For applications not falling under the categories above, the NA shall include the overall score and
overall comments on the application.
------------------------------------------
9