Is It User-Friendly or Google-Friendly?
In what became the most-discussed post of the day, a mystery author argued yesterday on TechCrunch for regulating Google’s algorithm for ranking search results and ads. The author compares the Internet to a vast, wonderful country — if the Internet were like Italy or Hawaii, and could only be visited two weeks a year, how many of us would vacation there? — whose borders are entirely controlled by Google, this inscrutable, Kafkaesque power.
But the author sort of misses the point. He worries that Google is using its power to screw Yahoo, AOL or Microsoft (where he probably works), not the consumer. He talks about sinister scenarios where Google arbitrarily refuses to run an ad, or dongs companies that don’t advertise from its search results. These scenarios either don’t seem likely or worrisome to me.
And more to the point, we don’t have to imagine such abuses of power; the Internet has already changed in literally billions of ways to accommodate Google. Overwhelmingly, Google has made the Internet a more organized, consumer-friendly place. But any mass of that size operating at the center of the Internet, however sunny and life-giving it may be, can’t help but deform it in unnatural ways, too.
This is a concern we first raised in April, arguing in an essay about how newspapers compete in the Google age that Google’s ability to shape the Internet is far more complete and long-lasting than Microsoft’s supposed monopoly of the desktop. I hope Bing will begin to change that, though I have been less impressed than others by its 8% jump in traffic, simply because some of that has been driven by one-time press hits and ads.
The fact is that every day, people make web pages dumber so Google can index them. At last Thursday’s Naked Truth, Jonathan Sposato talked about how Picnik is largely invisible to Google because Picnik is coded in Flash, a platform that allows you to do magical things to photos from a web browser – but one which Google’s robots still don’t really understand. What this means is that Picnik in some ways is too smart for Google –and pays the price, getting very little very traffic from search. It should come as no surprise then that fewer and fewer web applications have followed Picnik’s suit, even though as consumers we often wish they would.
For our part, Redfin just had a debate on Friday about whether to optimize our site for consumers or for Google. At stake was how to provide home-buying tips within our search application. One option is as a link to a separate web page. The other option is in-line, so that users can click on a control to see more information within the context of their search. The first option is Google-friendly, because the indexing robot doesn’t know how to click for more info, it can mostly only follow links. The second option is user-friendly, because most consumers don’t want their search interrupted while another page loads.
I think we’ll end up choosing the user-friendly option this time, but really the conversation shouldn’t even be happening. And it happens all the time, at Redfin and everywhere else. Just last year, we argued about how many similar listings to show on a given listing web page; users would prefer to see five similar listings but Google doesn’t mind if we show 20 or 50. The more similar listings we show, the more probably can get indexed. Today, we show 10 similar listings.
The differences go beyond how information is presented. Google doesn’t like websites that charge money, or limit information only to registered users. To show some broker’s listings, Redfin has to register users. Google’s robots can’t sign up for a Redfin account, and never see or index that information.
Some websites, like ZipRealty, register users to see details on every listing, so that Zip can follow up with a sales call or email. And I am very happy to report that Google punishes them for it. This is one reason why Redfin has been growing traffic faster than Zip. And it’s one reason that newspapers haven’t been able to charge users money, because Google won’t index content that it can’t show for free. This makes me unhappy, because some information is worth paying for.
So to Google, you’re either a media company, a website that displays information like a media company, or you’re somewhat invisible. Another way of thinking about this is that Google forces every website to be the braun — providing different types of information in a simple format its robots can understand — while Google itself is the brain, acting as the central hub for people to find their way to information. When we optimize our site for Google, we are enthusiastically out-sourcing more and more of our brain. Another way of saying this is that as more and more people access the listings on Redfin’s site via Google search, our home page is increasingly not a Redfin page, but a Google search page.
In many ways, I think this is a good thing, as it gives consumers the freedom to move from site to site using Google as a well-known guide. But we should at least acknowledge that all websites are now building themselves to be understood by one search engine. If you have to choose one company as the least common denominator for the entire Internet, you could no better than Google, but it is after all still only one company.
Vladimir said:
“or dongs companies that don’t advertise from its search results”
How do you dong a company, and what’s it like?
July 14, 2009 10:56 PM
Yumio said:
One thing to take into account here is that its not that Google’s robots “can’t” follow links that are behind a paywall (there are many workarounds), or they “can’t” index content that’s embedded in a hide/display mode ajax – its that engineers that determine robot behavior have “decided” not to.
What’s really worrisome is that Google’s engineers decide on these things due to a certain worldview/philosophy about the Web – that info should be free – that if everyone can’t get to it, it shouldn’t be part of Google’s organic search – that hide/display ajax are more often than not used maliciously by websites, so they have chosen to penalize content inside those javascripts.
Having worked for a big-company search engine (Yahoo!), I know from experience that its a very conscious choice by PEOPLE at Google – and to a large part by engineers with a specific worldview that is shaping how the Web is used/accessed, and its certainly a very disturbing trend.
July 15, 2009 1:10 AM
Glenn Kelman said:
@Vladimir: maybe I should have said ding; i just meant that there was a concern that companies would be punished.
@Yumio, couldn’t agree more about the Google worldview that content should be free. Where are you these days, if not at Yahoo?
Thanks to you both for such thoughtful comments…
July 15, 2009 11:14 AM
Anonymous Coward said:
Glenn said:
(the anonymous author) “worries that Google is using its power to screw Yahoo, AOL or Microsoft (where he probably works), not the consumer.”
Really? You think a Microsoft exec would anonymously criticize Google out of fear that an on-the-record comment would cause Google to retaliate by dropping Microsoft sites in Google’s search results? Are you nuts? Microsoft would LOVE to see Google try that.
My primary frustration with your blog is your continuing trickle of unwarranted, Silicon-Valley elitist, paranoid cheap shots at Microsoft. From your “Worked at Microsoft? Don’t bother applying here.” job-postings to your latest anonymous author conspiracy-theory, you seem to resort to Microsoft-bashing out of reflex (or laziness) when you don’t have anything intelligent to say. Without Microsoft, there wouldn’t be a Redfin today. Bite the hand that feeds you much?
July 15, 2009 4:12 PM
Glenn Kelman said:
Hi there anonymous!
I meant that the author of that article probably works for one of the three companies listed, not for Microsoft per se.
This doesn’t seem like such a stretch. The TechCrunch article itself said that the author works for a company hosting “one of the largest Internet sites” so Yahoo, AOL or Microsoft seemed to be the natural candidates; the article also said that the author preferred anonymity for fear of retaliation. Of the three, my guess is that the author works for Yahoo, whose employees tend to be more active on the blogs; I included AOL and Microsoft for completeness. If you read the TechCrunch article, I think you’ll see that I’m not drawing some bizarre conclusion about a conspiracy theory but rather summarizing what the author said: his concern is for the companies advertising on Google and competing with Google, not for the users of those websites.
As for my comment from a year ago about not wanting to hire someone in marketing who had worked “too long at Microsoft, Amazon or an agency,” I think I’ve made it clear that the only issue is this: folks with a long tenure in marketing at any very large company sometimes face a difficult transition when working for a startup that spends no money on marketing but still hopes to make a name for itself.
In that post, I emphasized that most of our
engineers and product managers come from Microsoft, and that many of my friends work there, so I really was only talking about marketing, which I think is very different at Microsoft, Amazon, Google or any other very large company as compared to Redfin. Marketing with and without money or leverage are different disciplines. I have no idea how to run a large advertising campaign, manage communications with a large number of partners, or think about how branding works across product lines and international borders.
Overall, I just have a hard time seeing how this article challenging Google is reflexive, lazy Microsoft bashing. Did you notice that I called out Bing’s marketshare and said that I hope it wins more? Or that the overwhelming thrust of this article is to call out some of the problems created by Google’s dominance in Internet search?
Is this article about Google really what set you off or is there some other issue? Are there other examples of anti-Microsoft bias? How am I a Silicon Valley elitist or paranoid? And good grief, why be anonymous? I don’t have any way of retaliating and wouldn’t if I could. We’re happy to have you here on this blog.
July 15, 2009 5:32 PM
Some Googler said:
I think you raise some interesting points, but IMHO (and admittedly biased) opinion, they’re dulled somewhat because you’re singling out Google, not (more correctly) talking about search engines overall.
Do Microsoft or Yahoo or Ask do a better job indexing Flash? Is Googlebot alone in the searchbot world in being sometimes a bit challenged in a “web20″ world? Even taking marketshare into account, do you really envision that things would be different if, say, Yahoo still ruled the search roost?
And re: the engineer-worldview comments, sorry, not buying it. If Google were to regularly index pages that are only available for pay, can you imagine the outcry from users… feeling tricked into clicking on links for content that they’d need to open up their wallets to see? Sounds like a customer/user-focus to me, not a philosophical issue.
Still, though, your larger point does strike me as more valid. Sites do have to sometimes make UI choices — and tough ones, at that — to balance user preferences against search engine current crawling and indexing capabilities. My guess is, though, that this’ll get better, not worse in the long run.
July 17, 2009 12:46 AM
Glenn Kelman said:
@Some Googler: I definitely agree that I’m holding Google to a higher standard. With great power comes great responsibility… but thanks for being so fair-minded and thoughtful about it. And thanks for leaving a comment. We’re big fans of the technologies that Google (and Microsoft!) create.
July 17, 2009 4:16 AM
Some Googler said:
Glen, a pleasure. I enjoyed reading your post, and I’m glad that people still hold us to a higher standard, even if I get admittedly a bit grumpy about it in the short term sometimes.
July 17, 2009 8:08 PM
smasmatry said:
oh my god. hehe
July 25, 2009 2:02 PM
Are Listing Agents Hurting Their Clients by Hiding Addresses? | Redfin Corporate Blog said:
[...] because most real estate websites are so spammy. And it has become increasingly clear that a web page requiring registration is invisible to Google — the Great Traffic Director in the Sky — whose indexing robots have no way of [...]
August 10, 2009 4:21 PM