This is an HTML version of an attachment to the Freedom of Information request 'Meetings with FairSearch'.



Ref. Ares(2018)2554324 - 16/05/2018
Ref. Ares(2018)2751838 - 28/05/2018
 
Review of the likely effects of Google’s proposed Commitments dated October 21, 2013 
(“Second Commitments”) 

December 9, 2013 
Professors David J. Franklyn ("Franklyn") and David A. Hyman ("Hyman")1 
1.  Introduction 
In May, 2012, the European Commission published its preliminary conclusion that Google limits 
choice in Internet search by (among other things) giving Google’s vertical search services 
(including Google Shopping and Google Hotels) more prominent page placement and links than 
it gives to its rivals in these areas.  On April 25, 2013, in its first set of proposed voluntary 
commitments (hereinafter Google’s “First Commitments”), Google proposed to resolve these 
concerns by providing more prominent placement for selected competitors’ websites (“three rival 
links,”) and by adding a label and disclosure that would indicate which services are and are not 
provided by Google.2  The stated purpose of these proposals is to give consumers an effective 
choice among vertical search services offered by Google and its rivals.  
We submitted an expert report in this matter on July 1, 2013, in which we analyzed the impact of 
the First Commitments.  Based on market studies that we conducted in the U.K., we found that 
the First Commitments, if enacted, would not address the Commission’s stated concerns and 
materially restore competition in the vertical search markets at issue.  The European Commission 
did not accept the First Commitments.   
On October 21, 2013, Google submitted a second set of revised commitments (hereinafter 
Google’s “Second Commitments”) to the European Commission.  On October 28, 2013, 
Google’s Second Commitments were released to various parties, including FairSearch Europe, 
for comment.  The modifications include, but are not limited to, certain enlargements of the three 
rival links box, changes in the display of the three rival links for mobile devices, and changes to 
the language of the proposed disclosure pop-up box that is supposed to enable consumers to 
clearly distinguish between paid and unpaid links, and between Google’s specialized (vertical) 
search results (including Google Shopping and Google Hotels) and Google’s generic (unpaid) 
search results.   
We were retained by Clifford Chance, advisors to FairSearch Europe (hereinafter “Fairsearch”), 
to conduct market tests on the proposed changes, and to render opinions as to the likely impact of 
Google's Second Commitments proposals on Internet users/shoppers.   Specifically, we were 
                                                             

 Experts’ background and qualifications are detailed in an Appendix.  A copy of Franklyn's Curriculum 
Vitae is attached as Exhibit A.   A copy of Hyman’s Curriculum Vitae is attached as Exhibit B.  Franklyn and 
Hyman have written two law review articles about consumer perceptions and the Internet, including an article 
published in Spring, 2013 in the Harvard Journal of Law and Technology, and a forthcoming article on search bias 
and search neutrality. Copies of these articles are attached as Exhibits C & D. 

 See April 25, 2013 Google Commitment Proposal.   
 

 
asked to test the visibility and consumer understanding of Google's Second Commitments 
proposals. 
We designed and conducted three online surveys in the United Kingdom that measured the 
likelihood that consumers would click on any of Google’s Second Commitments three rival 
links.  We tested the likelihood of consumer click through on images similar to those that would 
appear on non-mobile and on mobile devices.  If consumers do not click on the three rival links, 
then Google’s proposed remedy would not be meaningfully visible to consumers or likely to 
reflect the natural choices of consumers--and therefore would not be likely to restore competition 
in these areas, and maximize consumer welfare.  
The survey also measured whether respondents recognized and understood Google’s modified 
label and disclosure.  If consumers do not recognize and understand Google’s label and 
disclosure, then they would be even less likely to exercise a meaningful choice between 
Google’s vertical search services and the same services offered by Google’s competitors.  
As explained more fully below, we found Google’s Second Commitments are not likely to 
materially increase or restore consumer choice or competition in the vertical search markets at 
issue.  Google’s Second Commitments do not achieve the Commission’s stated objective of 
materially increasing the visibility of rivals and the awareness of those rivals by consumers.  
Specifically, we found that:   
A.  Google is the first stop for the overwhelming majority of users that employ a search 
engine to locate products and hotels. 
B.  The “three rival links” gather only a modest number of consumer clicks, both in absolute 
terms and relative to clicks on Google Shopping and Google Hotels. 
C.  Incremental increases in consumer attention that may accrue under the Second 
Commitments are unstable and easily destroyed by changes to Google’s search results 
page. 
D.  Presenting rival links in a competitively neutral fashion is a relatively straightforward 
matter.  Parity of rival link presentation improves consumer welfare.     
E.  Consumer confusion persists as to the difference between Google’s vertical search results 
and other search results.  Instead of reducing consumer confusion about search results, 
Google’s proposed disclosure/explanation materially increases confusion. 
II.  Survey Methodology 
Between November 12 and November 20, 2013, we completed three on-line surveys of a total of 
approximately 3,500 respondents residing in the United Kingdom.3  More specifically, using a 
series of simulated searches, the surveys examined: 
                                                             

 The survey was administered by Survey Sampling International (SSI) to a demographically representative 
sample of Google users in the United Kingdom.  We excluded from the analysis surveys that were not completed 
from an IP address in the UK.   
 


 
•  Whether the Second Commitments resulted in a material number of clicks on any 
of the proposed rival links;  
•  The effect of other modification to search page layout and labeling on click 
patterns;   
•  The effectiveness of the disclosure accompanying the rival links in 
communicating basic information.   
To examine these issues, we designed a series of multi-part interactive Internet surveys.   The 
surveys are attached as Addendum A.  The results presented in this report are based on 
respondents who completed the entire survey (with no missing results), and took more than 5 
minutes to do so.  We were left with a total sample of approximately 2,500 completed, useable 
respondents, fairly evenly divided across all three surveys.4   For the combined results, the 
margin of error on the figures we report is +2% for a 95% confidence interval.  For any given 
survey, the margin of error is roughly +3.5% for a 95% confidence interval.     
Each survey began with two questions designed to measure participants’ general online search 
behavior.  In Survey 1, we then asked respondents to run two sets of searches – one for an Apple 
iPod and the other for a Nikon camera.  In Survey 2, we asked respondents to run two sets of 
searches – one for an Apple iPod and the other for a Razer headset.  In Survey 3, we asked 
respondents to search for a hotel in Madrid.   
We chose these three searches in order to test three hypothetical sample product searches (here 
iPod, Nikon camera, and Razer), and one hypothetical locational search (here a search for a hotel 
in Madrid).  Respondents were asked to click on the link they would have selected if they were 
trying to buy an iPod, Nikon camera, or Razer headset, or trying to find a hotel in Madrid.   For 
each survey, respondents first viewed a “native” page of search results (i.e., the search results 
that would have appeared if an actual Google search had been run without the proposed 
Commitments being implemented) followed by a series of modified search results, reflecting 
Google’s proposed Commitments, and several variations we developed.  
In order to advance through the survey, respondents had to repeatedly run the search query, and 
then click through on whatever link they selected.   Respondents were not allowed to go back to 
an earlier question after progressing past it.  Because respondents had to click on something to 
advance, our results represent an upper-bound on clicks that would occur in a real-world search 
environment, where users can simply close the page without clicking on anything.   
After viewing the Google native page for each of these searches, respondents saw search results 
pages incorporating the three rival links specified in the Second Commitments.  For the three 
rival links for the products search, we chose well-known companies that would qualify for 
inclusion under the terms of Google’s Second Commitments; namely, Kelkoo, Pricerunner and 
Bizrate.  These are the same three rival links we used in our July, 2013 market test of Google’s 
First Commitments.  
                                                             

 See Addendum B, Table 1.  We separately analyzed the results for the excluded surveys, and found 
virtually identical results.  Thus, our exclusion criteria do not affect our findings.   
 


 
To validate our findings, we also tested the mobile version of rival links specified in the Second 
Commitments, along with multiple variations in which we modified the page layout and labeling 
of the search results page.5   Respondents were presented with these variations after re-running 
various Google searches.  Respondents were asked to click on the site where they were most 
likely to find what they were looking for, i.e., an iPod, Nikon camera, Razer headset or a hotel in 
Madrid. 
After completing these tasks, we showed respondents a series of screen shots of actual search 
results and asked them a variety of questions to determine whether they could tell the difference 
between the results in various regions of a typical search results page – that is, if they could 
correctly differentiate between paid and unpaid regions of the page and also identify how certain 
type of results were generated.   We also asked respondents to view Google’s proposed 
disclosure/explanation regarding its specialized vertical search results and asked respondents a 
series of questions to test their understanding of the proposed disclosure/explanation.  We 
concluded with a series of questions about the demographics of those completing the survey.  
III. Findings 
 
A.  Google is the first stop for the overwhelming majority of users that employ a search 
engine to locate products and hotels 
We asked respondents how they would go about locating a specific product (iPod) or a hotel in a 
city they didn’t know.  Of those who responded that they used a search engine to perform these 
tasks, Google accounted for 95% of respondents searching for an iPod, and 94% of those 
searching for a hotel in a city they didn’t know.  We obtained comparable results when we asked 
a similar question previously (compare Tables 2A and 2B in our initial report). 
B.  The three rival links in the Second Commitments gather only a modest number of 
consumer clicks, both in absolute terms, and relative to clicks on Google Shopping 
and Google Hotels
. 

We tested the three rival links proposed in the Second Commitments.6  Table 1A presents the 
results for Products, and Table 1B presents the results for Hotels. 
As Table 1A shows, in a product search, the three rival links garnered 0.9%-2.9% of total clicks, 
compared to 36.7% of total clicks on Google Shopping.7  
                                                             

 Our goal in testing these variations was to provide a baseline for assessing the impact of layout and 
labeling on consumer search behavior.  Because our primary task in this engagement was to test the proposals in the 
Second Commitments, we focus on them in this report.  Copies of the images that are referenced in this report are 
attached as Addendum C. 

 These variations are based on the images contained in Google’s Second Commitments. 
7  
By “Google Shopping” we are referring to all clicks in the specified region, and not simply the blue link at 
the top of the region, “Shop for ipod on Google.”  
The Second Commitments also authorize the shading of the region containing the rival links.  We 
 


 
Table 1A: Click Rates on Second Commitments Proposal – Non-mobile Product Searches 
Type of Link 
 
Pricerunner 
2.9% 
Rival Links 
Kelkoo 
1.2% 
Bizrate 
0.9% 
Google Shopping 
36.7% 
All Vertical Search Options 
41.8% 
All Other Link Types 
58.2% 
Total 
100% 
Pricerunner 
7.0% 
Share of All Vertical 
Kelkoo 
3.0% 
Search Options 
 Bizrate 
2.3% 
 
Google Shopping 
87.7% 
 
For ease of comparison, the bottom four rows in Table 1A (“Share of All Vertical Search 
Options”) compute the percentage of clicks on each of the three rival links and Google Shopping 
compared to the total clicks on all vertical search options on the search results page.8  This figure 
is effectively the “market share” for each of the specified rival links and for Google Shopping if 
we assume the Second Commitments are implemented.  As these rows indicate, in a product 
search, any given rival link secured 2.3%-7% of vertical search clicks, compared to 87.7% of 
vertical search clicks for Google Shopping.  
                                                                                                                                                                                                    
separately tested a shaded region, and found a modest increase in the absolute number of clicks on the three rival 
links (from 5% to 7.7%), and a reduction in the number of clicks on Google Shopping (from 36.7% to 28.4%).  In 
combination, this meant that Google Shopping’s share of all vertical search options declined slightly, from 87.7% to 
78.7%.  Further testing will be necessary to determine whether this finding is robust.   

We define clicks on all vertical search options as the sum of clicks on Google Shopping, Pricerunner, 
Kelkoo and Bizrate.  
 


 
Table 1B presents similar information for our hotel search.   
Table 1B: Click Rates on Second Commitments Proposal – Non-mobile Hotel Searches 
Type of Link 
 
Expedia 
3.8% 
Rival Links 
Priceline 
0.9% 
Trivago 
2.3% 
Google Hotels + Places 
28.1% 
All Vertical Search Options 
35.1% 
All Other Link Types 
64.9% 
Total 
100% 
Expedia 
10.9% 
Share of All Vertical Search 
Priceline 
2.5% 
Options  
Trivago 
6.7% 
 
Google Hotels + Places 
79.9% 
 
Table 1B shows that in a hotel search, the three rival links garnered 0.9%-3.8% of total clicks, 
compared to 28.1% of total clicks on Google’s vertical search options (Google Hotels + Google 
Places.9  As in Table 1A, the bottom four rows in Table 1B compute the percentage of clicks on 
each of the three rival links compared to the total clicks on all vertical search options on the 
search results page.10  As these rows indicate, in a hotel search, any given rival link secured 
2.5%-10.9% of vertical search clicks, compared to 79.9% of vertical search clicks on Google’s 
vertical search options.11  
Table 2A presents comparable results for mobile search images for Products, and Table 2B 
presents the same information for Hotels.  We tested two versions: the “Plain” version (which 
has the text “Other Sites” in the lower right hand corner of the Shopping and Hotels boxes), and 
a “Pop-up” version (which adds to the Plain version a large notification balloon that contains the 
text “Tap here for additional product (or hotel) search services,” along with a blue button that has 
the word “OK” in it).12   
                                                             

 By “Google Hotels,” we are referring to all clicks in the specified region, and not simply the blue link at 
the top of the region, “Hotels in Madrid, Spain on Google.”     
10 
We define clicks on vertical search options as the sum of clicks on Google Hotels, Google Places, Expedia, 
Priceline, and Trivago.  
11  
The Second Commitments also authorize the shading of the region containing the rival links.  We 
separately tested a shaded region, and found a modest increase in the absolute number of clicks on the three rival 
links (from 7% to 9.3%), and a slight reduction in the number of clicks on Google Hotels & Places (from 28.1% to 
27.6%).  In combination, this meant that Google Hotels & Places share of all vertical search options declined 
slightly, from 79.9% to 74.8%.  Further testing will be necessary to determine whether this finding is robust.   
12 
 These variations are based on the images contained in Google’s Second Commitments.   
 


 
Table 2A: Click Rates on Second Commitments Proposal – Mobile Product Searches 
 

Commitments Variation 
Plain 
Pop-up 
Type of Link 
 
 
“Other Sites” Link 
0.3% 
0.2% 
Google Shopping 
43.3% 
33.8% 
All Vertical Search Options 
43.6% 
34% 
All Other Link Types 
56.4% 
66% 
Total 
100% 
100% 
Share of All Vertical  “Other Sites” Links 
0.7% 
0.6% 
Search Options 
Google Shopping 
99.3% 
99.4% 
 
Table 2B: Click Rates on Second Commitments Proposal – Mobile Hotel Searches 
Commitments Variation 
Plain 
Pop-up 
Type of Link 
 
 
“Other Sites” Links 
0.0% 
0.1% 
Google Hotels + Places 
29.4% 
29.5% 
All Vertical Search Options (Including Places) 
29.4% 
29.6% 
All Other Link Types (Excluding Places) 
70.6% 
70.4% 
Total 
100% 
100% 
Share of All Vertical 
“Other Sites” Links 
0.0% 
0.4% 
Search Options 
Google Hotels + Places  100.0% 
99.6% 
 
As Tables 2A and 2B reflect, when we test the Mobile version of the Second Commitments, 
there are effectively no clicks on the “Other Sites” links in any of the variations.  For ease of 
comparison, the bottom two rows in Tables 2A & 2B (“Share of All Vertical Search Options”) 
compute the percentage of clicks on the “Other Sites” links and Google’s vertical search options 
compared to the total clicks on all vertical search options on the search results page.13  As this 
row indicates, in the Mobile setting, “Other Sites” links account for 0.6%-0.7% of clicks for 
products (Table 2A), and 0.0%-0.3% of clicks for hotels (Table 2B).  (We treat the actual pop-up 
notification balloon as an “Other Link Type) in both Tables 2A and 2B, since it only appears 
once, and is not, in fact, a vertical search link/option. 
 
One further complication: in mobile applications, a user must first click on the “Other sites” link, 
and then select a particular rival link on which to click.  We test only respondents’ initial click 
through, so our results reflect the maximum percentage of clicks on all rival links combined – 
and that only if we (implausibly) assume that no users click back or simply close the search 
results page.  
Tables 1A & 1B demonstrate that the percentage of non-mobile clicks on rival links is only 
                                                             
13 
We define clicks on all vertical search options on product search results pages as the sum of clicks on 
Google Shopping, Pricerunner, Kelkoo and Bizrate.  We define clicks on vertical search options on hotel search 
results pages as the sum of clicks on Google Hotels, Google Places, Expedia, Priceline, and Trivago.  
 


 
modestly higher than in our market test of Google’s First Commitments (there, it was never more 
than 2%).  However, click through rates for rival links in the Second Commitments continue to 
lag far behind click through rates for Google’s own vertical search results.  And, in the mobile 
context (Tables 2A & 2B), none of the versions we tested attracted more than a trivial number of 
clicks, even when the pop-up notification balloon was present, thereby calling attention to the 
“Other sites” link.14   
C.  Incremental increases in consumer attention that may accrue from the proposed 
rival links are unstable and easily destroyed by changes to Google’s search results 
page.  

We found that any incremental increases in consumer attention that might arise as a result of 
enhanced presentation of rival links in the Second Commitments is subject to dissipation if 
Google were to introduce new search features, specifically a search feature that it has already 
introduced elsewhere (e.g., in the United States).  This search feature is allowable under the 
Second Commitments.   
Specifically, we presented respondents with a page of search results that contained the following 
banner-strip at the top of the page of search results that included the three rival links as proposed 
in the Commitments: 
 
 
                                                             
14 
 In our test of Google’s First Commitments, we tested the proposed text for mobile uses, but we used a full-
size screen image.  In our test of Google’s Second Commitments for mobile uses, we used an image that looked like 
a mobile screen.  We found comparable click through rates in the earlier report  (0.1%-0.8%) versus our findings in 
this report (0.0%-0.3% for the plain images for Product and Hotel searches).  The first report did not contain a pop-
up image, so direct comparison of the percentage of clicks on the “Other sites” link for that image is not possible.      
 



 
 
Once we added this banner-strip (which we treated as part of the Google Shopping/Hotels 
region), click-through rates on shaded rival links on Products searches dropped from 5.1% to 
3.4% for products, and from 7% to 4.9% for hotels.  Click through rates on the Google Shopping 
and Google Hotels + Places regions also increased dramatically.  In combination, this meant the 
share of clicks on rival links dropped from 12.3% to 6.5% for products, and from 20.1% to 
10.4% for hotels.  Thus, in both relative and absolute terms, the introduction of the banner-strip 
largely negated the additional click-through effect of the Second Commitments (as compared to 
the First Commitments).  In unreported results, we were able to reduce the absolute and relative 
share of clicks on the rival links further still, by adding a floating pop-up that said “Search on 
Google Shopping” (or “Search on Google Travel”) to the search results page with the banner-
strip.  The floating pop-up is also allowable under the Second Commitments.    
D.   Presenting rival links in a competitively neutral fashion is a relatively 
straightforward matter.  Parity of Rival Link Presentation Improves Consumer 
Welfare 

We also tested two variations that we created, to determine how difficult it would be to present 
search results in a way that would substantially increase consumer attention and click-throughs 
on rival sites.    
In one alternative, we presented respondents with a page of search results that eliminated the 
Google Shopping or Google Hotels boxes (along with the proposed Commitments three rival 
 



 
links box) – and replaced those boxes with a single shaded box.  The box was headed “Compare 
prices on,” and had icons for four vertical search offerings.  The new box contained only icons 
and names of the vertical search offerings, and some abbreviated text describing the products 
that were available.  Thus, the four vertical search offerings were placed on an even playing field 
(including equal visual footing) with one another.  The image below shows an example of the 
box from the Razer product search, but we tested this approach in all three surveys.   
 
 
 We presented respondents with two versions of this box: one in which Google appeared in the 
box as the left-most icon/vertical search offering, and one in which Google appeared in the box 
as the right-most icon/vertical search offering.  This allows us to more effectively isolate the 
extent to which observed click through patterns are attributable to various factors (including 
search results page architecture, visually rich appearance of Google Shopping and Google 
Hotels, and the strength of the individual brands).  
Using this approach with a product search, we found that respondents clicked on the 
individual rival links 3.1% to 9.2% of the time, and clicked on Google Shopping 10.6% of the 
time when it was the left-most icon/vertical search offering, and 4.3% of the time when it was 
the right-most icon/vertical search offering.15  With a hotel search, we found that respondents 
clicked on the individual rival links 1.0% to 8.4% of the time, and clicked on Google Icon 6.4% 
of the time when it was the left-most icon/vertical search offering, and 3.2% of the time when it 
was the right-most icon/vertical search offering.16  Thus, when we create parity of appearance 
and placement, we obtain substantially higher click through rates in both relative and absolute 
terms for the rival links, reflecting the greater visibility of the rival links compared to the status 
quo.17  And, click through rates on Google Shopping and Google Hotels drop dramatically, 
indicating that position on the page and visually rich appearance is an important component in 
                                                             
15 If we look at click-through rates for each of the listed icons, we find Pricerunner received 9.2% of clicks when it 
was the left-most image, and 6.1% when it was the 2nd image from the left.  Kelkoo received 3.1% of clicks when it 
was 2nd image from the left, and 3.7% when it was the 2nd image from the right.  Bizrate received 4.7% of clicks 
when it was the 3rd image from the left, and 3.5% when it was the right-most image.   
16 If we look at click-through rates for each of the listed icons, we find Expedia received 8.4% of clicks when it was 
the left-most image, and 5.7% when it was the 2nd image from the left.  Priceline received 2.7% of clicks when it 
was 2nd image from the left, and 1.2% when it was the 2nd image from the right.  Trivago received 2.0% of clicks 
when it was the 3rd image from the left, and 1.0% when it was the right-most image.         
17 Viewed collectively, the three rival links/icons secured roughly 32% of clicks on vertical search options when 
Google was the left-most icon, and 53% of clicks on vertical search options when was Google was the right-most 
icon.  The results were virtually identical for product searches and hotel searches.  These “market share” figures are 
substantially greater than those we observe when we test Google’s Second Commitments.  Compare Tables 1A & 
1B    
 
10 


 
the dominance of Google Shopping and Google Hotels.  Finally, the fact that click through rates 
on Google Shopping and Google Hotels drop dramatically when it is in the right-most position 
(compared to the left-most position) further confirm that position on the search results page, (i.e., 
the architecture of the search results page, which gives Google’s vertical search offerings greater 
visibility and presence than competitors) rather than simply the strength of the brands of the 
competing vertical search providers, is responsible for an appreciable proportion of observed 
behavior. 
We found a similar effect when we tested respondents’ responses to an alternative variation, 
where we split the Google Shopping box into two equal-sized and equal-looking boxes.  One box 
was labeled Google Shopping Results and the other box was labeled “Compare results on 
Kelkoo.”  Both boxes contained the word “Sponsored,” and included two images of the products 
in question, along with some text.  This alternative variation is illustrated below: 
 
We tested this variation with Google Shopping on the right and on the left.  We found that parity 
of presentation resulted in an increase in clicks on Kelkoo when it occupied the left-side box 
(increasing to 12.2%), compared to the right-side box (6.9%).  Clicks on Google Shopping were 
23.2% when it occupied the left-side box, and 16.3% when it occupied the right-side box.  Thus, 
both Kelkoo and Google experienced significant increases in clicks when they were on the left 
side than when they were on the right side -- even though everything else remained the same.  
These findings show that click through rates on Google’s specialized search results are driven in 
part by the prominent position on the search results page and the more visually attractive (larger 
and photo-inclusive) presentation of Google’s own specialized search results.18  
We find the highest degree of rival link visibility and resulting consumer attention--and rough 
parity in click through rates --when rival links are displayed in a manner that is comparable to 
the manner in which Google’s own specialized search results are displayed.  The First and 
Second Commitments do not achieve this objective.  Our findings provide some insight into the 
factors that affect click through rates on vertical search offerings.   
                                                             
18 Unsurprisingly, we also observe a brand effect.  Google has higher absolute clicks than Kelkoo, irrespective of 
position.  But, the more significant issue is that visibility makes a difference, even after we control for brand effects.  
Google is as strong a brand when it is on the left side as when it is on the right side.  But, Google’s share of 
combined clicks on vertical search options is 77% when it is on the left side, and only 57% when it is on the right 
side.    
 
11 

 
E.  Consumer confusion persists as to the difference between Google’s specialized 
search results and other search results – and the disclosure actually increases 
confusion 

The Second Commitments provide that searchers who roll-over an icon that currently appears 
next to the word “Sponsored” at the top right of Google’s vertical will see a pop-up disclosure 
statement.  The statement is intended to clarify the origin and type of search results they are 
seeing.19  In unreported results, we asked respondents whether they had ever clicked on the icon 
that appeared next to the word “Sponsored” to see what would pop-up.  Only 12%-17% of 
respondents responded that they had ever done so.   
We also presented all respondents with the pop-up disclosure statement, and asked whether the 
statement improved their understanding of why content appeared on the search results page, 
made them more confused, or had no effect.  Table 3 presents the results.   
Table 3: Impact of Google's Proposed Pop-up Disclosure on Understanding of Why 
Content Appears 

 Type of Search 
Product 
Hotel 
Improved understanding 
27% 
28% 
More Confused 
43% 
44% 
No Effect on understanding 
30% 
27% 
Table 3 shows that of those who actually read the disclosure statement, only 27%-28% thought it 
improved their understanding of why content appeared on the search results page.   
We also showed respondents images that reflected the impact of the Second 
Commitments on the search results page (similar to those we used to generate the results in Table 
1).  But, these search results pages had each section of the webpage labeled, as “Region A,” 
“Region B,” and so on.  We then tested whether respondents were able to correctly explain the 
reason why links appeared in particular regions, such as Google Shopping, the Rival Links Box, 
and Algorithmic content.  We provided seven possible explanations: unpaid algorithmic content; 
links for which the listed companies pay to appear; price comparison service offered by Google; 
price comparison service offered by independent third parties; a fanciful control (unpaid links 
selected by Google’s Special Marketing Team); Other; and Don’t Know/Not Sure.20  
                                                             
19 
 The specific disclosure text reads as follows: “This link is inserted to show more results from Google’s 
Shopping results.  For shopping results from other relevant providers, click on the relevant links to other search sites 
below or see Google’s other search results.  Ads are based on your current search term.  Visit Google’s Ads 
Preferences Manager to learn more, block specific advertisers, or opt out of personalized ads.” 
20 
 Because of the nature of the search results page, more than one explanation might be correct for some of 
the regions/links.  For example, a respondent could correctly answer that links in the Google Shopping region were 
there because Google had been paid for the link to appear, or because it was a price comparison service offered by 
Google.  Similarly, for the rival links box, a respondent could correctly answer that links in that region were there 
because Google had been paid for the link to appear, or because it was a price comparison service offered by 
independent 3rd parties.  For these regions, we treated both of the listed answers as correct.   
 
12 

 
We found little evidence that respondents effectively differentiated paid from unpaid 
content, or understood that Google Shopping was offering vertical search results.  Only 51% of 
respondents correctly responded that the content in Google Shopping represented either paid 
content or a price comparison service offered by Google, with responses split evenly among 
these two choices.  This means that 48% of respondents did not understand why links appeared 
in the Google Shopping box – with responses split among those who thought it was unpaid 
algorithmic content (14%); those who picked our fanciful Google Special Marketing Team 
control (18%); those who thought it was a price comparison service offered by independent 3rd 
parties (6%); and those who didn't know, weren’t sure, or picked other (11%).  We obtained 
similarly mixed results when we asked respondents why algorithmic links appeared (only 41.5% 
correctly responded that these links were selected by Google’s computer formula and were 
unpaid), and why the rival links appeared (only 51% correctly responded that these links were 
either paid content or a price comparison service offered by independent third parties).   
Finally, we evaluated the impact of the disclosure statement on respondents’ understanding of 
why links appeared in the Google Shopping region.  We did this by asking them the same 
question twice – once before and then again after they had reviewed the disclosure statement.  
We find evidence that the disclosure statement actually reduced respondents’ understanding of 
the issue.  In the first set of surveys, the percentage of respondents answering that the links that 
appeared in Google Shopping were the result of a price comparison service offered by Google 
declined from 26.8% to 14.4%.  The percentage of respondents who didn’t know or were unsure 
increased from 10.1% to 17.2%, and the percentage that selected our fanciful control (i.e., links 
selected by the Special Marketing Team) increased from 17.8% to 22.1%.   
Stated differently, after being exposed to the disclosure statement, the percentage of respondents 
selecting a correct answer declined from 51.5% to 40.5%.  We obtain comparable results for the 
Hotels survey; after being exposed to the disclosure statement, the percentage of respondents 
selecting a correct answer declined from 50.4% to 41.8%.     
Appendix B includes more detailed information on all of these results.  Addendum D contains 
information on respondent demographics, including self-reported knowledge of search page 
layout and labeling.   
 
 
Summary/Conclusions: 
 
•  The proposed three rival links do not materially increase consumer awareness or the 
visibility of the proposed links to large majorities of consumers. 
•  Large majorities of consumers do not understand the origin of the rival links or their 
relationship to Google's other links. 
•  We found only slightly higher click-through rates for the Second Commitment proposals 
than we found for the First Commitment proposals.  We do not believe that the Second 
Commitments are likely to command materially increased consumer attention or restore 
competition for rivals that seek to compete with either Google Shopping or Google 
 
13 

 
Hotels and Places. 
•  We found that the proposal rival link remedy is unstable.  Modest changes to the 
appearance of the search results page, such as the addition of a banner-strip similar to the 
one currently being used in the U.S., eliminate the observed slight increases in visibility 
and prominence (as measured by the click-through rates on rival links).   
•  Google Shopping commands a high degree of consumer clicks and attention in part 
because of its location on the search results page, and the visually-rich display (e.g., 
pictures and mini-boxes within a box).  Google Hotels and Places benefit from similar 
treatment.  Confirming our earlier work, we find that page layout and design is more 
important than search result labeling in determining visibility and click-through rates.  
•  We find evidence that consumer preferences reflect both the strength of the underlying 
brands, as well as the way in which those brands are presented on the search results page.  
Absent parity of presentation, an appreciable number of consumers will be diverted from 
their preferred destination, which will reduce consumer welfare.  Conversely, consumer 
welfare is enhanced when artificial impediments to consumer choice are removed and 
rival sites are positioned in a manner that is visibly equivalent to the positioning and 
layout of Google’s own specialized search results.      
•  Modifications that do not materially affect page layout are unlikely to change preexisting 
competitive dynamics in search.  Modifications in page layout may or may not have an 
impact, depending on their design attributes.  Stated differently, we find the highest 
degree of consumer attention and competition for clicks when rival links are displayed in 
a manner that is comparable to the manner in which Google Shopping/Hotels results are 
displayed.   
•  We find high levels of consumer confusion as to the source and type of search results and 
a low level of ability to correctly differentiate between paid and unpaid results, or 
between Google’s vertical search and generic search results. 
•  We find that the proposed Second Commitment disclosure statement does not effectively 
communicate the necessary information (i.e., whether the region in question is paid v. 
unpaid, and the location and significance of the “rival links.”)  Indeed, the proposed 
language confuses many respondents.   
 
In summary, we find that the proposals offered by Google in the Second Commitments are 
unlikely to materially increase consumer attention, offer consumers meaningful choices or 
restore competition (as measured by click rates) in this space.  By testing various alternatives, we 
identified several factors that are more likely to draw consumer attention to rival links, offer 
consumers meaningful choices and thereby improve competition (again, as measured by click 
rates) in this space.  We also found that Google’s proposed label and disclosure – which is aimed 
at improving consumers’ ability to differentiate Google search results from rivals’ search results 
– are actually confusing or misleading to a substantial number of consumers.   
 
 
 
 
Submitted this 9th day of December, 2013 
 
14 

 
Professors David J. Franklyn & David A. Hyman 
 
15 

 
Appendix  
Franklyn is a tenured, full professor of law, specializing in trademark law, at the University of 
San Francisco School of Law (“USF”).  Franklyn is the Executive Director of the McCarthy 
Institute for Intellectual Property and Technology Law at USF and Director of the Center for the 
Empirical Study of Trademark Law (CEST).  The McCarthy Institute, of which Franklyn is 
Executive Director, conducts wide-ranging empirical research on consumer perception issues in 
both the United States and Europe.  Franklyn is also the Director of the Masters of Law Program 
for U.S. and foreign lawyers in Intellectual Property Law at USF.  Franklyn teaches and writes 
primarily about trademark law and consumer perception issues.  Franklyn is editor-in-chief and 
co-author of McCarthy’s Desk Encyclopedia of Intellectual Property Law. 
Franklyn works in collaboration with Professor McCarthy on a number of projects, including his 
treatise on trademark law.  Professor McCarthy is the author of a seven-volume treatise on 
trademark law.  It is the most-widely cited treatise on trademark law in the United States, having 
been cited by courts in over 3,500 judicial opinions, including in a recent decision by the United 
States Supreme Court in a case in which Franklyn was retained as an expert.  
Franklyn has consulted and/or served as an expert witness on behalf of numerous clients in 
numerous cases involving consumer perception issues, including in matters in the United States, 
Asia, the European Union, the Middle East and South America.  Franklyn has been retained as 
an expert in several matters involving the issue of consumer perceptions.  Franklyn was recently 
was retained by the International Olympic Committee to provide trademark advice and advice 
regarding the consumer impact of various logos regarding the 2016 Olympic Games.  Franklyn is 
a licensed attorney in Illinois and California.  Franklyn has written several law review articles 
about consumer perceptions on the Internet.  Franklyn has also acted as a consultant to several 
clients in matters related to trademark law, consumer perceptions and the Internet. 
Hyman is the H. Ross and Helen Workman Chair in Law and Professor of Medicine at the 
University of Illinois.  He is tenured in both the College of Law and the College of Medicine.  
He heads the Epstein Program in Health Law and Policy at the University of Illinois College of 
Law.  He teaches and writes about empirical law and economics, principally involving the 
regulation of health care, and competition law and policy.  He is the author of more than 100 
articles in student edited law reviews, and peer-reviewed medical, health policy, and law 
journals.  
In competition law and policy, Professor Hyman served as Special Counsel to the Federal Trade 
Commission from 2001-2004.  In that capacity, Professor Hyman was principal author and 
project leader for the first joint report ever issued by the Federal Trade Commission and 
Department of Justice, “Improving Health Care: A Dose of Competition” (2004).  He has also 
published numerous other articles involving competition law and policy, including a series of 
articles with Professor William Kovacic, the former Chairman of the United States Federal Trade 
Commission.   
Hyman has been deposed in various cases involving health care fraud and pharmaceutical 
pricing, and has testified in one case involving pharmaceutical pricing.  
 
16