Every few months, we’ve tested Google Maps against Virtual Earth, and the result was always the same: Google’s script and tiles loaded considerably faster, but Virtual Earth was as much as four times faster at adding a ton of items to the map.  Since our user interface can add up to 500 houses at a time to a map, we just felt like Google wasn’t able to give us the performance we needed.  To be fair, part of VE’s speed was due to a bulk add feature that we had lobbied for them to put in, but it worked well, and so we put Google aside, wistfully looking at those speedy script and tile load times.

A few months ago, though, we started contract renegotiations with Microsoft, and we decided to give Google Maps a closer look.  One of my colleagues, the brilliant Dan Fabulich of Selenium fame, figured out that we could code our own custom GOverlay to make Google Maps display items much faster than it had previously.  The question then became: how hard would it be to port our site from one platform to the other?  And would it be worth it to do so?

To answer those questions, I took off from work one Friday determined to see what a meth-fueled* weekend of coding could come up with.  I started demolition of the codebase Friday night and worked the rest of the weekend from wake until sleep to get something up and running.  By the end of the weekend, I had a working prototype that had most of the major features of our site.

The most shocking thing to me was how relatively easy the transition was, but that turned out mostly to be a testament to how little of the map API we actually use.  There are a lot of features of Virtual Earth that we’ve never touched: traffic, 3D, directions, geocoding, and the native zoom and pan controls, just to name a few.  All we do is tell the map to go somewhere, tell it to draw some houses on the map, and once in a while tell it to draw a polygon representing a neighborhood.  In return, we expect the map to send us an event when the user drags or pans it.  Both VE and Google handle these basic needs deftly, and their APIs aren’t wildly different.  Most notably, Google Maps and Virtual Earth use the same numbering scheme and heights for their zoom levels, so we didn’t have to worry about URLs with zoom levels suddenly looking wrong on Google.  The most complicated parts of our application turn out to be things that have nothing to do with the map: parsing massive data structures we retrieve via XHR, locking the UI at certain moments to prevent the user dragging mid-search, providing back button support, and making sure we load our page resources in the most network-efficient way.

So, how did it feel as a programmer to move from VE to Google?  Overall, it was great; there were a bunch of times where it felt like Google’s Map API just worked better or was better organized than VE’s.  None of them were large issues, but they added up.  As examples:

• Google provides both a movestart event and a dragstart event so that you can easily distinguish between map moves that are caused by programmatic calls to move the map versus map moves that are caused by the user.  Doing this in VE has often proved painful for us.
• Fewer things happen in a timeout in Google Maps, which was a pattern that caused us considerable pain in VE.
• Resizing seems like less heavyweight an operation in Google Maps, as the map doesn’t move by default when you resize.
• Google seems more respectful of the global Javascript and CSS namespaces than VE.  VE, for example, adds methods to Object on some browsers, which breaks for-in loops.  They also define a style that has a selector of just “a img”.  What’s worse: that style has a !important on it, so overriding it is non-trivial.

There were of course a few issues with Google Maps (including at least one troubling bug we found in both maps.google.com and the maps API), but the overall experience was silky smooth.  And when we did come across problems, we found that, while the Google API docs were less thorough than comparable VE docs, there were many, many more forum and blog posts to help us find the answer.

After my weekend code-a-thon, I put the GMaps prototype up on a test server in our QA labs, and Dan ran a few of our standard performance tests against it.  The first thing he found was the map page consistently took 2.5 seconds faster to fire its onload event, seemingly due to the size and CDN-ness of the GMaps script.  Next, he ran our absolute worst case scenario test (driven by Selenium, naturally) where we add 500 houses to the map.  In IE6, we saw a speedup of about 10 seconds.  Even on a fairly fast browser like Firefox 3, the speedup was 3 seconds.  And that was in addition to the 2.5 second load speedup.

On the web, times like that are just gold; there’s just no way you can ignore a several second speed up.  Even though we had a lot of i’s to dot and t’s to cross, I knew at that moment we were going to make the transition.

My prototype seemed about half done and had taken two days to code; it took another two weeks of the search team’s time to get it up to production quality.  The difference between a product that was coded in a weekend and basically works and one that is coded and tested and works reliably for hundreds of thousands of people is massive.  As Glenn is always reminding us, 95% done is at best half done.

Anyway, hope you all like the end result, and I hope you find using Redfin that much snappier.  My thanks to all the folks who worked on the transition, especially Dan, Navtej, Jane, and Jim.  If you have any questions about the transition, feel free to ask in the comments.

* If you’re reading, Mom, no, it wasn’t literally meth-fueled.

Backing up: for the last few days, we’ve been revamping our registration and login process at Redfin to make it easier for users, and I must admit that what we’ve found has depressed me to no end. We want to put an inline form that lets users log in without ever leaving the page, but it turns out that it’s very difficult to do this reasonably securely. Much more depressing has been the realization that many sites that I use daily, sites I respect, have abysmal security around login and registration. Our awesome summer intern Aditya said after a long discussion of the myriad ways to hack the login process: “Man, I’m never banking online again.” He then confessed that he’d never banked offline (and didn’t know how), which made me feel impossibly old.

The topic that most scared Aditya was “Man In The Middle” attacks and the impossibility of providing a reasonably secure experience from an HTTP page. The problem is this: suppose someone evil has set up a wifi router in your favorite coffee house and that he has control over that router. Because all of your traffic is going through that wifi router, Mr. Evil can not only listen to every packet that is transmitted between you and the Internet, he can also modify any packet. Now, on an HTTPS page, that’s not a problem because (a) the packets that the router can see are encrypted and (b) the browser will alert the user with a big scary warning if the any of the packets have been modified along the way. HTTP pages, on the other hand, will act completely normally if Mr. Evil mucks around with them.

What this means is that any form on an HTTP page is inherently unsafe, even if it posts to an HTTPS page.

Let me repeat that: ANY form on an HTTP page is inherently unsafe.

Why? Because as a user, you have no idea if the form at http://awesomesite.com came from awesomesite.com or from Mr. Evil’s nefarious lair. And if the form did come from Mr. Evil, chances are that it will send your credit card number back to Mr. Evil (and your username and your password and your SSN and your mother’s maiden name…). Worse, it could send the credit card info to Mr. Evil before continuing and sending it along to awesomesite.com. Neither you nor awesomesite.com would be any the wiser.

Many web developers mistakenly believe that an HTTP form (on http://awesomesite.com) that posts to an SSL URL (https://awesomesite.com) is safe, because the communication channel for the post is secure between client and server. But Mr. Evil can change http://awesomesite.com freely, so he can make the form post to anywhere he wants. This is why we tell users again and again rule #1 of web security: Never enter confidential information, especially credit card numbers, unless you see the SSL lock in your browser.

Which brings us back to Adgregate Markets. Adgregate is a company that launched at the TechCrunch 50 yesterday, and they have a product that creates interactive banner ads that allow the user to complete a shopping checkout process completely inside an ad, without leaving the host page at all. They say that this both increases conversion rates, which is a Good Thing for advertisers, and increases page stickiness, which is a Good Thing for publishers. The panel at TC50 reacted positively to Adgregate, and the fact that their business model takes a cut of every sale that happens through their ads is certainly attractive.

There’s a problem here, though, and it’s packaged neatly in Adgregate’s marketing copy: “provide(s) secure transactions on both secure and non-secure pages.” As I said above, I think this simply isn’t possible. If the host page is HTTP, then Mr. Evil can swap out Adgregate’s Flash ad for an ad that looks exactly like it but sends all of the info to Mr. Evil’s server, not to Adgregate. Users will be none the wiser until a laser death ray shows up on their MasterCard statement.

On the other hand, maybe Adgregate has figured out how to securely conduct a conversation on an HTTP page. Many, many smart people have tried for over a decade to solve this problem and failed, but it’s possible Adgregate cracked that nut. If they have, though, it would arguably be a better business model to patent and sell the intellectual property. There are a lot of people who would pay good money for it.

As I write this post, Adgregate’s running third in a prediction market for winning best in show. My amateur prediction is that they won’t win, but that they will probably come out of TC50 with a good boost in publicity. My fear, though, is that they will start to find success with their “secure” forms on HTTP pages, and that they will succeed at convincing users that those forms are safe. (To be fair, Adgregate is far from the only company that is attempting to convince users of the safety of HTTP forms, a fact we discovered in the course of our research on inline registration. To my horror, I found out that even my bank encourages users to enter their password into a non-SSL page.)

If users become inured to entering confidential info on to HTTP pages, then everyone in the consumer web space loses. Because when those Adgregate ads get hacked (and if they become at all successful, they will be hacked), people will lose some degree of faith in the consumer web. And after Mr. Evil has gotten you once, it’s hard to believe that he won’t get you again.

(Dr. Evil pose photo used under Creative Commons from flickr user caitlinator.)

This is when we got well and truly confused. It looked like the browser was downloading our CSS, because items were appearing in the right place on the page, but all color had drained out of the world. Was Firefox in some sort of a funk? At first, we guessed that perhaps the user had some sort of ad blocker that was blocking images from our content delivery network, but we quickly noticed that some images made it through. What was going on here? Solution after the jump.

On closer inspection, it became clear that all of the images that were failing were actually background-images set in CSS; images added to the page as normal IMG tags were working fine. Unfortunately, we use CSS sprites in our site wherever we can to speed up performance, which means that the vast majority of the images in our UI were invisible to the user.

The next question was: why on earth would this be happening? Well, it turns out that Firefox has a preference that allows a user to override the colors specified by websites and use a default color scheme the user wants. This preference not only affects text color, but it also eliminates all background-images on the page.

A week ago we released a new version of Redfin with a bunch of fixes, one of which was that we moved the icon for houses on the map into a CSS sprite, which made the icon disappear for the poor user who had this pref set in Firefox. After we gave the user instructions on how to turn off the preference, it worked like a charm. (For the record, you turn the pref off by going to Tools –> Options in Windows XP or Firefox –> Preferences… in Mac OS X, then Content Tab –> Colors… –> Check “Allow pages to choose their own colors, instead of my selections above”.)

As programmers, what should we do here? I’ll admit I haven’t thought through a good solution yet, and we don’t want to abandon sprites outright for performance reasons. One thought I’ve had (but not tested yet): perhaps we could use JavaScript to test if the background-image properties are not getting set and in that case replace the critical background-imaged elements with normal image tags. It doesn’t sound like much fun, and it’s probably too small a use case to worry about, but it’s worth a try.

Thoughts about this problem? Solutions? Let me know in the comments.

Last January, we ported Redfin from a proprietary map platform to Microsoft’s Virtual Earth, a move that has enabled us to move faster in adding new markets and features. We’ve gotten both praise and criticism for the move over the last 9 months, but no piece of feedback has been as constant or as vociferous as the cry of Mac users who want us to support Safari.

As a MacBook Pro user myself, I understand the desire for us to support the native Mac browser (of course, we do support Firefox on OS X), but Virtual Earth didn’t support Safari, so we were pretty much out of luck. And although I like Safari, we have to admit that Safari 2 is buggy at best with support for CSS and JavaScript. Applications like Redfin or Virtual Earth that use a lot of client-side scripting are bound to have problems with Safari 2. This doesn’t mean that it’s impossible to support Safari 2, but it is likely to require significant testing and engineering to get a site to work properly.

The upcoming Safari 3, though, has taken a massive leap forward in terms of standards support, and it’s crazy fast to boot. While we don’t support Safari yet officially, you can use Safari 3 with Redfin if you trick Redfin into thinking that you are running Firefox, and it Mostly Works ™.

Before you start, please note that Safari 3 is in beta, it is not to be used on mission-critical systems, this is all at your own risk, your computer could spontaneously explode, etc. With that disclaimer out of the way, here’s how you get Safari and Redfin to play nice:

1. Download a copy of Safari 3, either the Safari 3 Public Beta from Apple or the WebKit nightly from webkit.org. The Safari Public Beta is a better tested build, but it may not have all the bug fixes of the nightly build from webkit.org. Also note that installing the Safari 3 Public Beta will get rid of Safari 2 on OS X, whereas I believe (but am not 100% sure) that the WebKit nightly can co-exist with Safari 2.
2. Turn on debug functionality for Safari. On Macs, do it this way:
   1. Quit Safari so it’s not running.
   2. Run the program called “Terminal” which is located in your Applications->Utilities folder.
   3. Type the following command into Terminal exactly as shown:

      defaults write com.apple.Safari IncludeDebugMenu 1

   4. Quit from the Terminal program, and launch Safari again. Safari should now have a “Debug” menu. If not, check the command you typed above. Case is important.

   On Windows boxes, turn on the debug menu this way:

   1. Quit Safari so it’s not running.
   2. Open the file at:

      c:Documents and Settingsyour usernameApplication Data
      
      Apple ComputerSafariPreferences.plist

   3. Add the following text to the file before the final </dict>:

      <key>IncludeDebugMenu</key>
      
      <true/>

   4. Launch Safari again. Safari should now have a “Debug” menu.
3. Launch Safari 3. In the Debug menu, select User-Agent –> Firefox 2.0.x.x. (The exact version of Firefox in the Debug menu depends on which build of Safari you have.)
4. Go to www.redfin.com, and use Redfin to your heart’s content.

There are a few things to note here. First, you will have to switch the User Agent every time you open a new tab or restart Safari. Second, there are also a few known bugs we’ve seen with just the limited testing we’ve done:

1. Depending on which version of Safari 3 you get, the images of houses next to the map may be stretched weirdly.
2. The zoom bar on the map isn’t centered correctly.
3. Keyboard shortcuts on the map page (up/down/left/right) don’t work correctly.

If you try this out and find any other bugs (and we’re sure they’re there), let us know in the comments section.

(Photo credit: p_c_w on Flickr)

We’ve been really trying to hammer on our map page performance for the last few weeks, and it became clear early on that performance of our current grid is suffering when there are a lot of houses on the map. As with many of our performance problems, it is especially pronounced in Internet Explorer 6. (Sidenote: Glenn recently told a focus group of customers that I would personally pay them $20 each to upgrade from Internet Explorer 6 or switch to Firefox, and I’m beginning to come around to the idea despite the current state of my bank account.)

The most frustrating part for me personally is that the solution for slow performing list controls is fairly well understood and exists in practically every major window library: virtual list controls. A virtual list control is one in which you don’t add all the items to the list immediately on creation but rather hand the list control a callback function instead. The list then only asks for the handful of rows it needs to display at any one moment and can therefore scale up to thousands or millions of rows. In pseudo-code, that looks something like this:


function initList() {
var list = new VirtualList();
list.setRowCount(1000);
list.setRowContent(contentCallback);
}


function contentCallback(rowIndex) {
return "this is the content for row #" + rowIndex;
}

TurboGrid uses the virtual list strategy, and it reacts much more quickly than our current grid control. Our next release will use TurboGrid to speed up the list, and we’ve been very pleased with both its performance and its feature set. So why am I frustrated? Well, it’s mostly because I learned about virtual lists in my first few months of professional coding after college in (ahem) 1997. And while browser-based AJAX development is a lot of fun, it can be disheartening when you are re-solving problems that you thought were solved over a decade ago.

In the meantime, though, we’re happy we found out about the TurboGrid open source project from the good folks at Sitepen and were able to help in some small way to make it a reality. Until we all have jetpacks, flying cars, and a built-in grid control in HTML, TurboGrid will help keep us chugging along, and we hope that users will notice the speed up (although that $20 is still on the table for IE6 users).

(Photo credit: samjuk on Flickr; Title credit: Daniel H. Wilson’s Where’s My Jetpack?)