Is It User-Friendly or Google-Friendly?

In what became the most-discussed post of the day, a mystery author argued yesterday on TechCrunch for regulating Google’s algorithm for ranking search results and ads. The author compares the Internet to a vast, wonderful country — if the Internet were like Italy or Hawaii, and could only be visited two weeks a year, how many of us would vacation there? — whose borders are entirely controlled by Google, this inscrutable, Kafkaesque power.
But the author sort of misses the point. He worries that Google is using its power to screw Yahoo, AOL or Microsoft (where he probably works), not the consumer. He talks about sinister scenarios where Google arbitrarily refuses to run an ad, or dongs companies that don’t advertise from its search results. These scenarios either don’t seem likely or worrisome to me.
And more to the point, we don’t have to imagine such abuses of power; the Internet has already changed in literally billions of ways to accommodate Google. Overwhelmingly, Google has made the Internet a more organized, consumer-friendly place. But any mass of that size operating at the center of the Internet, however sunny and life-giving it may be, can’t help but deform it in unnatural ways, too.
This is a concern we first raised in April, arguing in an essay about how newspapers compete in the Google age that Google’s ability to shape the Internet is far more complete and long-lasting than Microsoft’s supposed monopoly of the desktop. I hope Bing will begin to change that, though I have been less impressed than others by its 8% jump in traffic, simply because some of that has been driven by one-time press hits and ads.
The fact is that every day, people make web pages dumber so Google can index them. At last Thursday’s Naked Truth, Jonathan Sposato talked about how Picnik is largely invisible to Google because Picnik is coded in Flash, a platform that allows you to do magical things to photos from a web browser  — but one which Google’s robots still don’t really understand. What this means is that Picnik in some ways is too smart for Google –and pays the price, getting very little very traffic from search. It should come as no surprise then that fewer and fewer web applications have followed Picnik’s suit, even though as consumers we often wish they would.
For our part, Redfin just had a debate on Friday about whether to optimize our site for consumers or for Google. At stake was how to provide home-buying tips within our search application. One option is as a link to a separate web page. The other option is in-line, so that users can click on a control to see more information within the context of their search.  The first option is Google-friendly, because the indexing robot doesn’t know how to click for more info, it can mostly only follow links. The second option is user-friendly, because most consumers don’t want their search interrupted while another page loads.
I think we’ll end up choosing the user-friendly option this time, but really the conversation shouldn’t even be happening. And it happens all the time, at Redfin and everywhere else. Just last year, we argued about how many similar listings to show on a given listing web page; users would prefer to see five similar listings but Google doesn’t mind if we show 20 or 50. The more similar listings we show, the more probably can get indexed. Today, we show 10 similar listings.
The differences go beyond how information is presented. Google doesn’t like websites that charge money, or limit information only to registered users. To show some broker’s listings, Redfin has to register users. Google’s robots can’t sign up for a Redfin account, and never see or index that information.
Some websites, like ZipRealty, register users to see details on every listing, so that Zip can follow up with a sales call or email. And I am very happy to report that Google punishes them for it. This is one reason why Redfin has  been growing traffic faster than Zip. And it’s one reason that newspapers haven’t been able to charge users money, because Google won’t index content that it can’t show for free. This makes me unhappy, because some information is worth paying for.
So to Google, you’re either a media company, a website that displays information like a media company, or you’re somewhat invisible. Another way of thinking about this is that Google forces every website to be the braun — providing different types of information in a simple format its robots can understand — while Google itself is the brain, acting as the central hub for people to find their  way to information. When we optimize our site for Google, we are enthusiastically out-sourcing more and more of our brain. Another way of saying this is that as more and more people access the listings on Redfin’s site via Google search, our home page is increasingly not  a Redfin page, but a Google search page.
In many ways, I think this is a good thing, as it gives consumers the freedom to move from site to site using Google as a well-known guide. But we should at least acknowledge that all websites are now building themselves to be understood by one search engine. If you have to choose one company as the least common denominator for the entire Internet, you could no better than Google, but it is after all still only one company.