A Virtual Earth to Google Maps Transition: From Idea to Deployment In a Few Weeks

Whenever I out myself as a member of the Redfin search team to someone who has used Redfin, one of the first questions I get is, “so why do you guys use the Microsoft Map?  Why didn’t you choose Google?”.  The full answer is a bit long, but the short answer is easy: speed.

Every few months, we’ve tested Google Maps against Virtual Earth, and the result was always the same: Google’s script and tiles loaded considerably faster, but Virtual Earth was as much as four times faster at adding a ton of items to the map.  Since our user interface can add up to 500 houses at a time to a map, we just felt like Google wasn’t able to give us the performance we needed.  To be fair, part of VE’s speed was due to a bulk add feature that we had lobbied for them to put in, but it worked well, and so we put Google aside, wistfully looking at those speedy script and tile load times.

A few months ago, though, we started contract renegotiations with Microsoft, and we decided to give Google Maps a closer look.  One of my colleagues, the brilliant Dan Fabulich of Selenium fame, figured out that we could code our own custom GOverlay to make Google Maps display items much faster than it had previously.  The question then became: how hard would it be to port our site from one platform to the other?  And would it be worth it to do so? Read the rest of this entry »

Teaching users to get hacked, or why HTTP forms are bad for the web

This morning I was scanning TechCrunch while still in bed, and I saw a news item about a hot new Silicon Valley startup, Adgregate Markets, that has either solved a really vexing problem in online security or has made the situation a lot worse. Given what we’ve been working on recently at Redfin, I’m pretty sure it’s the latter.

Backing up: for the last few days, we’ve been revamping our registration and login process at Redfin to make it easier for users, and I must admit that what we’ve found has depressed me to no end. We want to put an inline form that lets users log in without ever leaving the page, but it turns out that it’s very difficult to do this reasonably securely. Much more depressing has been the realization that many sites that I use daily, sites I respect, have abysmal security around login and registration. Our awesome summer intern Aditya said after a long discussion of the myriad ways to hack the login process: “Man, I’m never banking online again.” He then confessed that he’d never banked offline (and didn’t know how), which made me feel impossibly old.

The topic that most scared Aditya was “Man In The Middle” attacks and the impossibility of providing a reasonably secure experience from an HTTP page. The problem is this: suppose someone evil has set up a wifi router in your favorite coffee house and that he has control over that router. Because all of your traffic is going through that wifi router, Mr. Evil can not only listen to every packet that is transmitted between you and the Internet, he can also modify any packet. Now, on an HTTPS page, that’s not a problem because (a) the packets that the router can see are encrypted and (b) the browser will alert the user with a big scary warning if the any of the packets have been modified along the way. HTTP pages, on the other hand, will act completely normally if Mr. Evil mucks around with them.

What this means is that any form on an HTTP page is inherently unsafe, even if it posts to an HTTPS page.

Let me repeat that: ANY form on an HTTP page is inherently unsafe.

Why? Because as a user, you have no idea if the form at http://awesomesite.com came from awesomesite.com or from Mr. Evil’s nefarious lair. And if the form did come from Mr. Evil, chances are that it will send your credit card number back to Mr. Evil (and your username and your password and your SSN and your mother’s maiden name…). Worse, it could send the credit card info to Mr. Evil before continuing and sending it along to awesomesite.com. Neither you nor awesomesite.com would be any the wiser.

Many web developers mistakenly believe that an HTTP form (on http://awesomesite.com) that posts to an SSL URL (https://awesomesite.com) is safe, because the communication channel for the post is secure between client and server. But Mr. Evil can change http://awesomesite.com freely, so he can make the form post to anywhere he wants. This is why we tell users again and again rule #1 of web security: Never enter confidential information, especially credit card numbers, unless you see the SSL lock in your browser.

Which brings us back to Adgregate Markets. Adgregate is a company that launched at the TechCrunch 50 yesterday, and they have a product that creates interactive banner ads that allow the user to complete a shopping checkout process completely inside an ad, without leaving the host page at all. They say that this both increases conversion rates, which is a Good Thing for advertisers, and increases page stickiness, which is a Good Thing for publishers. The panel at TC50 reacted positively to Adgregate, and the fact that their business model takes a cut of every sale that happens through their ads is certainly attractive.

There’s a problem here, though, and it’s packaged neatly in Adgregate’s marketing copy: “provide(s) secure transactions on both secure and non-secure pages.” As I said above, I think this simply isn’t possible. If the host page is HTTP, then Mr. Evil can swap out Adgregate’s Flash ad for an ad that looks exactly like it but sends all of the info to Mr. Evil’s server, not to Adgregate. Users will be none the wiser until a laser death ray shows up on their MasterCard statement.

On the other hand, maybe Adgregate has figured out how to securely conduct a conversation on an HTTP page. Many, many smart people have tried for over a decade to solve this problem and failed, but it’s possible Adgregate cracked that nut. If they have, though, it would arguably be a better business model to patent and sell the intellectual property. There are a lot of people who would pay good money for it.

As I write this post, Adgregate’s running third in a prediction market for winning best in show. My amateur prediction is that they won’t win, but that they will probably come out of TC50 with a good boost in publicity. My fear, though, is that they will start to find success with their “secure” forms on HTTP pages, and that they will succeed at convincing users that those forms are safe. (To be fair, Adgregate is far from the only company that is attempting to convince users of the safety of HTTP forms, a fact we discovered in the course of our research on inline registration. To my horror, I found out that even my bank encourages users to enter their password into a non-SSL page.)

If users become inured to entering confidential info on to HTTP pages, then everyone in the consumer web space loses. Because when those Adgregate ads get hacked (and if they become at all successful, they will be hacked), people will lose some degree of faith in the consumer web. And after Mr. Evil has gotten you once, it’s hard to believe that he won’t get you again.

(Dr. Evil pose photo used under Creative Commons from flickr user caitlinator.)

CSS Sprites + Firefox Content Preferences = Site Go Boom

I arrived at work today with a perplexing tech support e-mail in my inbox. A customer was claiming that although Redfin had never worked well in Firefox, last week we totally broke it. Needless to say, we were all a little surprised (and more than a little worried) over here, since we use Firefox all day every day. When we asked the user to send us screenshots, they looked something like this:

This is when we got well and truly confused. It looked like the browser was downloading our CSS, because items were appearing in the right place on the page, but all color had drained out of the world. Was Firefox in some sort of a funk? At first, we guessed that perhaps the user had some sort of ad blocker that was blocking images from our content delivery network, but we quickly noticed that some images made it through. What was going on here? Solution after the jump.

Read the rest of this entry »

Redfin Supports Safari! Um, kinda…

UPDATED: As of January 31, 2008, Redfin supports Safari 3 without any of the trickery described below. We’ll leave the post here for the curious, but in the meantime, there’s no reason to go through these steps on our account.

Last January, we ported Redfin from a proprietary map platform to Microsoft’s Virtual Earth, a move that has enabled us to move faster in adding new markets and features. We’ve gotten both praise and criticism for the move over the last 9 months, but no piece of feedback has been as constant or as vociferous as the cry of Mac users who want us to support Safari.

As a MacBook Pro user myself, I understand the desire for us to support the native Mac browser (of course, we do support Firefox on OS X), but Virtual Earth didn’t support Safari, so we were pretty much out of luck. And although I like Safari, we have to admit that Safari 2 is buggy at best with support for CSS and JavaScript. Applications like Redfin or Virtual Earth that use a lot of client-side scripting are bound to have problems with Safari 2. This doesn’t mean that it’s impossible to support Safari 2, but it is likely to require significant testing and engineering to get a site to work properly.

The upcoming Safari 3, though, has taken a massive leap forward in terms of standards support, and it’s crazy fast to boot. While we don’t support Safari yet officially, you can use Safari 3 with Redfin if you trick Redfin into thinking that you are running Firefox, and it Mostly Works ™.

Before you start, please note that Safari 3 is in beta, it is not to be used on mission-critical systems, this is all at your own risk, your computer could spontaneously explode, etc. With that disclaimer out of the way, here’s how you get Safari and Redfin to play nice:

1. Download a copy of Safari 3, either the Safari 3 Public Beta from Apple or the WebKit nightly from webkit.org. The Safari Public Beta is a better tested build, but it may not have all the bug fixes of the nightly build from webkit.org. Also note that installing the Safari 3 Public Beta will get rid of Safari 2 on OS X, whereas I believe (but am not 100% sure) that the WebKit nightly can co-exist with Safari 2.
2. Turn on debug functionality for Safari. On Macs, do it this way:
   1. Quit Safari so it’s not running.
   2. Run the program called “Terminal” which is located in your Applications->Utilities folder.
   3. Type the following command into Terminal exactly as shown:

      defaults write com.apple.Safari IncludeDebugMenu 1

   4. Quit from the Terminal program, and launch Safari again. Safari should now have a “Debug” menu. If not, check the command you typed above. Case is important.

   On Windows boxes, turn on the debug menu this way:

   1. Quit Safari so it’s not running.
   2. Open the file at:

      c:Documents and Settingsyour usernameApplication Data
      
      Apple ComputerSafariPreferences.plist

   3. Add the following text to the file before the final </dict>:

      <key>IncludeDebugMenu</key>
      
      <true/>

   4. Launch Safari again. Safari should now have a “Debug” menu.
3. Launch Safari 3. In the Debug menu, select User-Agent –> Firefox 2.0.x.x. (The exact version of Firefox in the Debug menu depends on which build of Safari you have.)
4. Go to www.redfin.com, and use Redfin to your heart’s content.

There are a few things to note here. First, you will have to switch the User Agent every time you open a new tab or restart Safari. Second, there are also a few known bugs we’ve seen with just the limited testing we’ve done:

1. Depending on which version of Safari 3 you get, the images of houses next to the map may be stretched weirdly.
2. The zoom bar on the map isn’t centered correctly.
3. Keyboard shortcuts on the map page (up/down/left/right) don’t work correctly.

If you try this out and find any other bugs (and we’re sure they’re there), let us know in the comments section.

(Photo credit: p_c_w on Flickr)

Where’s My Jetpack?

As was mentioned a few places on the web today, we’re proud to announce that Redfin has contributed to a project that will open source TurboGrid, the excellent JavaScript grid control made by the TurboAjax group.

We’ve been really trying to hammer on our map page performance for the last few weeks, and it became clear early on that performance of our current grid is suffering when there are a lot of houses on the map. As with many of our performance problems, it is especially pronounced in Internet Explorer 6. (Sidenote: Glenn recently told a focus group of customers that I would personally pay them $20 each to upgrade from Internet Explorer 6 or switch to Firefox, and I’m beginning to come around to the idea despite the current state of my bank account.)

The most frustrating part for me personally is that the solution for slow performing list controls is fairly well understood and exists in practically every major window library: virtual list controls. A virtual list control is one in which you don’t add all the items to the list immediately on creation but rather hand the list control a callback function instead. The list then only asks for the handful of rows it needs to display at any one moment and can therefore scale up to thousands or millions of rows. In pseudo-code, that looks something like this:


function initList() {
var list = new VirtualList();
list.setRowCount(1000);
list.setRowContent(contentCallback);
}


function contentCallback(rowIndex) {
return "this is the content for row #" + rowIndex;
}

TurboGrid uses the virtual list strategy, and it reacts much more quickly than our current grid control. Our next release will use TurboGrid to speed up the list, and we’ve been very pleased with both its performance and its feature set. So why am I frustrated? Well, it’s mostly because I learned about virtual lists in my first few months of professional coding after college in (ahem) 1997. And while browser-based AJAX development is a lot of fun, it can be disheartening when you are re-solving problems that you thought were solved over a decade ago.

In the meantime, though, we’re happy we found out about the TurboGrid open source project from the good folks at Sitepen and were able to help in some small way to make it a reality. Until we all have jetpacks, flying cars, and a built-in grid control in HTML, TurboGrid will help keep us chugging along, and we hope that users will notice the speed up (although that $20 is still on the table for IE6 users).

(Photo credit: samjuk on Flickr; Title credit: Daniel H. Wilson’s Where’s My Jetpack?)