DIRECTORATE MARKET OPERATIONS
ECB -
CONFIDENTIAL
DIRECTORATE FINANCIAL STABILITY
DIRECTORATE GENERAL ECONOMICS
DIRECTORATE GENERAL ADMINISTRATION
REFERENCE: 19051/F/2011
To:
Procurement Committee (GR Contract)
From:
Evaluation Team
Date:
Subject:
OLIVER WYMAN - BID EVALUATION SHEET
The following document provides a structure for evaluating the bids received from short-listed and
qualified investment banks.
The structure mirrors the criteria stated in the Request for Proposal with respect to eligibility, price, and
quality of proposal. Moreover, the document provides guidance with respect to scoring in each category.
Please utilize this sheet to ensure that we may compare each procurement team member’s assessment of
each component of the bid, and also so that we have documentation of our decision-making process.
The ECB shall evaluate all proposals using a scoring system with a maximum of 100 points. The
maximum number of points achievable per award criterion is equivalent to the percentage weightings
allocated to the award criteria. When the scorer has completed each section in this evaluation, please
provide the final sub-scores and total score (based on a scale of 100) and the name of the firm below:
RfP Evaluation Matrix
Name of Firm: Oliver Wyman
Weighting
Rating (0-
5)
Multiple
Subtotal Maximum
I. Price
4
x5
20
25
II.Quality
75
a. Proposal
4
x6
24
30
b. Team
4.5
x8
36
40
c. Contract
2
x1
2
5
Total
82
100
RESTRICTED
Part I. Eligibility Criteria
Firms may be disqualified from the tender process prior to the formal evaluation of their bid if they do not
meet basic criteria due to the following deficiencies:
• Significant conflict of interest that would not be sufficiently addressed through Chinese walls.
• Bid that is either excessively low and places inadequate resources to the assignment, or
excessively high such that the bid is well above the ceiling of EUR 3 million.
• The bid does not dedicate sufficient qualified senior resources or specialized teams.
• Other issues mentioned in the document (not submitted in English, late submission, etc.)
If disqualified, please state “Disqualified” and explanation below:
Part II. Assessment of the Price and Technical Bid
Assessment of the price and technical bid is includes a 0-5 scale rating in each subcategory, and is then
multiplied by a weighting multiple to achieve the allocated weightings. For example, quality of proposal
is weighted by x8, such that a high score of 5 will achieve (5x8=>) a maximum 40% weighting.
Note the obvious point that there are 6 possible scores in a 0-5 scale; as there will be 6 or less bidders at
this time, one could conceivably “rank” the bidders (best bid receives the highest score and no bidder
receives the same score). That said, if we feel that either price or components of quality are sufficiently
identical, we may give two equal ratings in a price or quality subcategory. However, it is recommended
that we strive to provide sufficient differentiation so as not to result in equal total ratings of two or more
bidders.
A.
Price
Price is given a 25% weighting in the total score. The Contribution relative to Price of the proposal shall
be evaluated such that the highest contribution relative to price will receive the full amount of points (5).
The lowest contribution may receive from zero to 2 points, depending on the variance of the contributions
between the worst bid and the top bid or several bids. The contributions of remaining bidders will be
scaled appropriately between the minimum and maximum points awarded.
It is suggested that the assignment of the lowest score reflects the extent to which it falls below the range
of scores. If the differentiation between the highest and lowest bidder is modest (e.g. all bidders provide
sufficient staffing for EUR3mm, or bids below EUR3mm offer only modest reductions) the bidder with
2
RESTRICTED
the least advantageous bid will not be unduly burdened by a 0 score
. It is suggested that the Procurement
Committee hold a brief discussion on this issue when the prices are reviewed.
Note that we must consider both price and contribution to price. All else equal, the highest contribution
to a fixed price would receive the highest score. However, if we receive a variety of contribution levels
(daily rates multiplied by number of aggregate working days), we will need to calculate a comparable
price/input metric.
Factors contributing to a higher score include:
• Total bid below EUR 3 million based on lower billing rates, with person day input equal to or
more than other bidders.
• All else equal, the composition of input is weighed toward senior staff, ensuring the ECB receives
senior counsel.
• Be careful not to simply reward firms for lower bids because their natural billing rate is low;
maximum score should go to the firm that provides best value for money. Firms with higher daily
rates – which can reflect quality and reputation in the market – can still receive a high score if the
contribution level is very high, or if they provide the ECB explicit discounts on the daily rate.
Factors contributing to a lower score include:
• Total bid at or above EUR 3 million and/or low contribution of aggregate staff input.
• A low bid but even lower staff input.
• The firm provides substantially less input from more experienced staff relative to other bidders.
Total Points Given (0-5): _____4________
Explanation of key reasons:
- Overall bid: Total fees amount to
Oliver
Wyman and
(a subcontractor).
The overall costs are however capped at
and
for Oliver Wyman and
,
respectively.
- Daily rate contribution: Daily weighted average fee is
for Oliver Wyman and
which is considered as fair.
- Input contribution: Oliver Wyman and
provide a reasonable staff input with adequate
seniority distribution.
3
RESTRICTED
B.
Quality of Proposal, Team and Presentation
The quality related award criteria listed in Section 4.5.2 shall be assessed one by one against the ECB’s
requirements set out in the specifications and evaluated on the basis of a scoring system from zero to five
points as follows, and scaled thereafter aligned with the maximum allowable points per subsection:
Score
Definition
0 point
No evidence/Total failure: The tender totally fails to address the criterion under
examination or cannot be assessed due to missing evidence.
1 point
Very Poor: The criterion under examination is addressed in an incomplete and
unsatisfactory manner; serious concerns.
2 points
Poor: The criterion under examination is partly addressed but with a few major
gaps or issues.
3 points
Fair: The criterion under examination is generally addressed with minor issues.
4 points
Good: The criterion under examination is fully addressed.
5 points
Excellent: The criterion under examination is fully addressed and the tender
offers some added value.
The points scored in the Quality Section of the RfP, parts (a), (b), and (c) will be scaled by a multiple of
six, eight, and one time respectively, to reflect the 30%, 40%, and 5% points attributed to this part of the
section.
(i)
Quality of the Proposed Approach and Plan in response to our outlined scope of work
The scope of work sought to
. Bidders were asked to provide
initial thoughts and an approach to addressing these topics related to
, based on past work, case
studies, and/or initial analysis.
Factors that may result in a high score include:
• Clear understanding of
.
• Draws on analysis of
and relevant case studies, demonstrating that the bidder can
integrate analysis and past experiences effectively to deliver initial advice.
•
• Demonstrates firm’s ability to assess these topics in a thoughtful, rationale, feasible, integrated
way
.
Factors that may result in a low score:
4
RESTRICTED
• Weak, cursory assessment based on generalities and obvious approaches.
• Indication that the firm does not have much experience in one or several areas within the scope,
and did not conduct sufficient analysis to overcome this experience gap
. [Note, team experience
will be judged separately, so in this section we will allow superior analysis to offset lack of
experience].
• Inability to integrate experience and knowledge to provide practical advice.
Several points to note. The score should include our assessment both of the written proposal and the
bidders presentation in Frankfurt. It may be the case that midlevel staff write the proposal, and senior
level staff deliver the presentation: where there is a difference between the quality of advise between the
two, we should weight our view on the advice given by members of the suggested core team, as they will
be our advisors on the assignment.
Total Points Given (0-5): ______4_____
Overall the proposal is very comprehensive providing a clear strategy how to approach the well identified
problems. It also provides some preliminary hypothesis concerning the roots of the problems suggests
solutions and associated policy recommendations. It demonstrates a very good initial understanding of the
situation of
banking system.
.
.
Explanation of key reasons:
5
RESTRICTED
(ii)
Quality of Team
The quality of team receives the highest weighting of any individual factor. Thus, several factors are
offered for consideration.
(a) Core team.
Factors that contribute to high rating of the core team include the following:
• Seasoned executive/senior members of team who have well-rounded experience in the
areas highlighted in the scope of work, and also have provided such advice to banks,
governments and/or central banks.
• Number of core team members and aggregate depth of experience in the relevant areas.
• Procurement committee’s comfort with the team, the manner in which the provide
advice, perceived credibility, etc.
Factors that contribute to low rating of the core team include the following:
• Executive/senior members of team do not have the requisite relevant experience, or the
experience is narrow, or not distributed well across the team. Care must be taken to avoid
a team that has a very seasoned executive but relatively inexperienced staff, as there is
always a risk that the most senior members of the core team may be less available.
• Small core team offered and not well rounded
• Procurement committee does not have the comfort that the team will develop the desired
working relationship, in which it appreciates a broad range of factors (systemic, ECB
mandate and constraints, etc) beyond technical issues.
• Perceived lack of commitment to the assignment.
6
RESTRICTED
(b) Specialized Team
Factors that contribute to high rating of the core team include the following:
• Depth of expertise and knowledge
•
•
Factors that contribute to low rating of the core team include the following:
• Insufficient depth of expertise of
•
.
• Evidence that specialized team would not be integrated with the core to provided sufficient
needed advice.
Subtotal Points Core (0-5): ______4.5________
Subtotal Points Specialized (0-5) ____4.5______
Total Points Team (avg.) _______4.5________
Explanation of key reasons:
- Core
The team comprises a good mix of expertise reflecting the combination of pools from Oliver
Wyman (strategic and risk consulting) and
). The challenge is for the two teams to integrate seamlessly. The team has
relevant experience in several relevant European countries
. Oliver Wyman’s team has a very broad and long experience in strategic advisory
services in particular in the financial sector.
- Specialized
While having a subcontractor
will raise coordination challenges, it may be an asset for
meeting the RfP’s objectives regarding technical issues such as
.
’s senior
7
RESTRICTED
advisor has useful experience in
financial sector, albeit this may raise issues related to conflict
of interests.
(iii)
Compliance with ECB Contractual Terms
The compliance with the ECB’s contractual terms and conditions (as set out in the draft contract attached
as Annex 3) shall be assessed on the basis of the changes or additions proposed by Tenderers. When
assessing the proposed changes or additions, the ECB shall take into account to which extent these
changes or additions after negatively the ECB’s interests, in particular its rights in case of non-
performance and the allocation of risks between the parties. The comments shall be evaluated on the basis
of a scoring system from zero to five points as follows:
Score
Definition
0 point
Tenderer does not accept the ECB’s draft contract and/or comes up with
its own draft contract.
1 point
Tenderer proposes several fundamental changes or additions to the ECB’s
draft contract.
2 points
Tenderer proposes one fundamental change/addition or several substantial
changes/additions to the ECB’s draft contract.
3 points
Tenderer proposes one substantial change to the ECB’s draft contract.
4 points
Tenderer proposes only minor changes to the ECB’s draft contract.
5 points
Tenderer accepts the ECB’s contractual terms and conditions without any
reservations. Clarifications or comments of editorial nature do not lead to
deductions.
Total Points Given (0-5): 2
Explanation of key reasons: Oliver Wyman requests ten amendments to the ECB’s contract terms. The
requested limitation of their liability must be considered as fundamental, and the limitations to the ECB’s
use of Oliver Wyman’s advice (clause 5) is substantial. Some amendments can be considered as mere
clarifications (contractor’s offices) or as acceptable in principle subject to clarification (subcontracting
(must be limited to the use of
); payment of services provided in case of termination (must
be subject to ECB claims for non- or malperformance); non solicitation). All comments should be further
discussed, clarified and negotiated if Oliver Wyman is invited to a bidder presentation. The initial
assessment was communicated to OW together with a mark up version of the contract, showing those
8
RESTRICTED
changes the ECB can accept, on 13 January 2012 with a request for reply on Monday, 16 January.
Following negotiations, the ECB and Oliver Wyman agreed on a revised version of the agreement that
incorporates some of their proposals. Their liability is limited for slight negligence and the ECB can use
the advice received and forward it to
(to private
parties only if they sign a ‘release letter’ that will be attached to the contract. With the final version the
contract score remains at 2 points as there are still some substantial changes compared to the ECB’s
standard contract but in view of the exceptional circumstances of this procurement, the limited
competition and the urgency of the advice DG-L considers the final version as acceptable for the ECB.
Part III. Procurement Committee Assessment leading to Final Calculations
The worksheets contained in this document will allow the assessor to provide remarks that support a
particular category score. However, before providing the final scores and total weighted score (on a scale
of 100) on the first page of this document, it is recommended that the assessor review each section across
the bidders for a fairness check. In particular, the assessor should review the ranking of scores across the
bidders compared to the written assessment, to ensure that the numeric scores fairly rank the bidders
based on the written justification for each. Once completed, the assessor may then provide the draft scores
for each bidder.
Thereafter, the Procurement Committee shall gather to assess their scores per section across bidders, and
diverging scores (i.e. those that are more than one point apart) should be briefly discussed by individuals
representing the diverging scores. The co-chairs of the committee will offer their opinions if, based on
this discussion, they believe the appropriate weighting should be different than the average of scores of
the individual team members.
Upon completion, the co-secretaries will draft a master appraisal for each firm that provided the final
subsection and total scores and key assessments for each score.
9