We’re All Newspapers Now

Redfin posted an essay on TechFlash late last night discussing the implications of what TechCrunch called “Danny Sullivan’s beautiful rant.” Sullivan argues that newspapers should just opt out of Google’s index if they don’t want their home pages to be replaced by Google News.

But, as is usually the case with posts like this, we disagree. Simply saying that we have to love all Google’s policies and products or leave its index ignores the dominant position Google has in shaping how people consume just about everything on the Internet.

The great responsibility that comes with that falls not just on Google but on all of us in technology who make it easier to distribute for free content that others depend on selling to make a living.

This is not to say that Google is to blame for having a business that profits from having as much content as possible be free on the Internet; most other technologies do the same, and the consumer benefits too, at least for now.

The problem is bigger than Google. The problem is the “beautiful rant” itself — a pervasive  belief that the newspapers, musicians, film-makers and story-tellers who create all this wonderful stuff on the Internet can only do so on our terms, which is to say  zero, nada, free.

I guess the writers and artists could go build their own search engine, their own music-sharing and video-sharing technologies, but wouldn’t we rather they keep writing “30 Rock,” reporting from Darfur and recording sappy love songs?

If we all become distributors because nobody pays creators, won’t the whole Internet become a pirate’s junkyard of free stuff?

We’re All Just Link Farmers Now…

For a long time, we have all brooded and marveled at how the entire Internet has been deformed by the enormous mass of Google at its center. Just last Friday, Google seemed to conclude that the size of the Web is no larger than the size of its index.

Websites aren’t built anymore for people to use, but for Google to index. As the New York Times noted earlier this month, every Google-optimized site is, in a way, a temple that reinforces Google’s power, built according to rules that Google, like some benign but distant god, has only vaguely outlined.

And let’s face it, without these rules, the Web would be a far less organized, interconnected place: Google favors websites built according to simple principles, whose pages can be updated by its audience, which get lots of links from other sites.

But any system can be exploited, and some popular sites have prospered in part by gaming the system: flooding the index with pages and creating fake links.

Sometimes at Redfin we try to figure out whether to spend our time building a good site for our customers, or for Google. Mostly we try to convince ourselves these are one and the same thing.

For example, we spent a few months re-mapping millions of URLs so they better describe the page they represent (such as http://www.redfin.com/WA/Seattle/2416-24th-Ave-E-98112/home/138344), a Google-driven feature that our users may appreciate but probably not as much as townhouse or parking filters.

And even though we’ve just scratched the surface of how we need to optimize our site for Google, we’ve recently begun to wonder — yes, this is heresy — if drawing lots of random visitors from Google should really be our goal.

Redfin isn’t like ad-driven sites, which make money from visitors regardless of whether they come from Google through some random carom or hear about the site from a friend and plunge in up to their eyeballs.

We don’t get paid until a customer uses our site to buy a house, which usually involves hundreds of visits over half a dozen months (hence our Freakish Depth strategy).

If Google sends the wrong people our way, our traffic may go through the roof but our business grows more slowly. Which is exactly what has been happening (that is, we’re growing, but not as fast as our traffic). So we’ve begun to wonder if Google’s visitors ever stick around long enough to become Redfin addicts.

So we formed a theory, that as the percentage of visitors coming from Google increased (excluding folks searching on “Redfin” and similar terms), more would bounce off. The table below maps some key stats through the first half of 2008; the “bounce rate” on the last row is how many visits end with only one page being visited.

As you can see, as more visits come from Google, fewer of them bounce off. Which basically tells us that Google not only sends more people our way, but sends people our way who apparently decide they’ve come to the right place. What a great search engine! So much for our theory that traffic from Google is lower quality.

In fact, when we survey Redfin’s visitors, 15% say they come to us via a Google search on a non-Redfin term, vs 18% of the people who ultimately buy a home through us. Which is just to say that I should have never doubted Google (traffic stats courtesy of, you guessed it, Google Analytics).

All the Homes for Sale (Well, Nearly All)

One Sunday morning last fall, my great friend Conan, a debonair former nose-tackle who often advises me to get in touch with my feminine side, called to ask about using our site. It took half an hour to convince someone who knows I would never lie to him in a million years to run a search on Redfin. And almost all that time was spent on one simple question: “do you have all the homes for sale?”

“Great question,” I said as real estate inventory has somehow become one of my favorite subjects, second only to “how popular I was in high school,” gossip, my many grudges, and what different drugs are like. “We have all the broker-listed homes for sale.”
“And that’s ALL the homes for sale?”
“Well, technically there’s always a few sold by the owner, without a broker. It’s called FSBO.”
“Yeah?” Conan said. “You don’t have those?”
“No.”
“Is that it?”
“And there’s also stuff in foreclosure, that’s been repo’d by the bank, before the bank has hired a broker to put it in the MLS.”
“Yeah?” Conan said. “Is THAT it?”
“And there’s always the possibility that you could sell your place to your cousin, without ever putting it on the market. We wouldn’t have that either.”
“So basically you’re saying you kind of suck,” Conan said. We had now come to very familiar ground in our 20-year friendship.
“Yeah,” I said. “But everyone else sucks too. Many suck worse.”
In the background, I could hear Conan typing in his first search.

A few minutes after our call ended, the U.S. real estate market collapsed, putting more inventory in a gray market of foreclosures. And we soon began working with I-don’t-have-an-office-number-just-a-cell-phone middle-men, data-scrapers and offshore aggregators to get our hands on every home for sale we could find, minus outdated garbage, copyrighted information, stuff that wasn’t really for sale, and teasers where consumers would have have to pay someone else for the address.

This culmination of this effort is the latest version of our site, live since 6 a.m. this morning, which has bank-owned foreclosures and for-sale-by-owner listings alongside all the listings from the Multiple Listing Service (MLS). It’s a little controversial, and lots of fun to play around with and of course we think it’s beautiful too.

Not that even now the new version of Redfin has all the homes for sale – we only had enough money to get data on foreclosures once the bank owns them, but not before, when they’re up for auction. For the markets we serve, we should have more real estate listings than anyone else. And the foreclosure data we do have is unique, because we give the foreclosures’ actual addresses and bank contact information, which other sites require you to pay for. Redfin is the only site we know of that aggregates bank-owned properties for free.

We’ve hit up more than 50 for-sale-by-owner sites for their inventory, with only two major gaps: Zillow and craigslist. Zillow is, we hope, coming soon. Someone offered us a craigslist feed, no questions asked. But when we emailed craigslist about it, Internet god Craig Newmark confirmed this was a violation of the terms of service. What was incredible about his reply was that it took exactly three minutes and it came from The Man Himself (Craig Newmark rocks).

On other fronts, we’ve fixed up our URLs for Google, and juiced up our MLS integration again, in both Boston and Southern California, so that we can get comprehensive inventory updates every 15 minutes (our only laggard MLS integration is now in Ventura County). While we were at it, we grabbed more open-house data, let people search for open houses, and upgraded our email notifications to tell people when their favorite listings change prices or sell. Next up, the data team is hoping to get photos of very-recent past sales.

But for now we’ve also made it more obvious how to search for data on past sales, a sweet feature that Redfin has always had but which no one ever knew about because it used to be hard to find. No longer! For the markets we serve, you can easily see what any home sold for between last week and 20 years ago (the lag is often more than a week, depending on how long it takes the transaction to record).

What does this all add up to? As our press release explains, the new version takes us deeper and deeper into Freakish Depth, which is our strategy to build the best real estate site by getting the best data. Because we’re a broker in the thick of doing deals, we have access to data as it’s being recorded. And unlike lead-generation sites, we want people to use Redfin to go all the way, getting everything they need to buy the home without having to talk to anyone or pay anything.

The strategy started in January with super-low-latency-MLS integration, bird’s-eye views, two new estimates, Excel integration, neighborhood boundaries and comparable listings. Then we got deeper with listing statistics, which analyzes where the traffic for a listing is coming from, and how inventory levels in the neighborhood have changed.

A gratifying subplot of all this is that the MLSs with whom we have sometimes jousted worked with us on this release. One MLS guided us on how we could show different types of inventory without violating the rules. And another provided the square footage data we’d been asking for. It’s still a delicate balance, sharing data among competitors, and working within a common set of rules to protect the different brokers’ clients.

Which makes us all the more careful not to screw things up. For years, we’ve wanted to add inventory to Redfin’s site, but have hesitated out of respect for the brokers who share their listing data with Redfin, many of whom have legitimate concerns about properties for sale directly from banks or owners. In the end what swayed us was:
a. the consumers who want above all else to see all the homes for sale,
b. the market, where in places like San Diego 40% of the sales have been foreclosure-related,
c. guidance from some of the MLSs we work with.

Hopefully, we’ve found a way to show consumers more information without causing any harm to other MLS members. We’re going to roll out flat-fee services in the next few weeks to work with buyers on for-sale-by-owner and foreclosure properties, but we won’t help anyone go around a listing broker if one has been hired.

Many thanks to all the engineers, product managers and customers – a San Francisco focus group talked us into buying the addresses for foreclosures — who came together to create this site. We worked awfully hard on it, some of us through occasionally daunting circumstances, and are anxious to see how you like it. We hope there aren’t too many bugs. And please, spread the word!

*PS: we may add auction properties in the future but in that case we would have to require users to pay an extra fee to see the address.