A Virtual Earth to Google Maps Transition: From Idea to Deployment In a Few Weeks

VE and GMaps, side-by-sideWhenever I out myself as a member of the Redfin search team to someone who has used Redfin, one of the first questions I get is, “so why do you guys use the Microsoft Map?  Why didn’t you choose Google?”.  The full answer is a bit long, but the short answer is easy: speed.

Every few months, we’ve tested Google Maps against Virtual Earth, and the result was always the same: Google’s script and tiles loaded considerably faster, but Virtual Earth was as much as four times faster at adding a ton of items to the map.  Since our user interface can add up to 500 houses at a time to a map, we just felt like Google wasn’t able to give us the performance we needed.  To be fair, part of VE’s speed was due to a bulk add feature that we had lobbied for them to put in, but it worked well, and so we put Google aside, wistfully looking at those speedy script and tile load times.

A few months ago, though, we started contract renegotiations with Microsoft, and we decided to give Google Maps a closer look.  One of my colleagues, the brilliant Dan Fabulich of Selenium fame, figured out that we could code our own custom GOverlay to make Google Maps display items much faster than it had previously.  The question then became: how hard would it be to port our site from one platform to the other?  And would it be worth it to do so?

To answer those questions, I took off from work one Friday determined to see what a meth-fueled* weekend of coding could come up with.  I started demolition of the codebase Friday night and worked the rest of the weekend from wake until sleep to get something up and running.  By the end of the weekend, I had a working prototype that had most of the major features of our site.

The most shocking thing to me was how relatively easy the transition was, but that turned out mostly to be a testament to how little of the map API we actually use.  There are a lot of features of Virtual Earth that we’ve never touched: traffic, 3D, directions, geocoding, and the native zoom and pan controls, just to name a few.  All we do is tell the map to go somewhere, tell it to draw some houses on the map, and once in a while tell it to draw a polygon representing a neighborhood.  In return, we expect the map to send us an event when the user drags or pans it.  Both VE and Google handle these basic needs deftly, and their APIs aren’t wildly different.  Most notably, Google Maps and Virtual Earth use the same numbering scheme and heights for their zoom levels, so we didn’t have to worry about URLs with zoom levels suddenly looking wrong on Google.  The most complicated parts of our application turn out to be things that have nothing to do with the map: parsing massive data structures we retrieve via XHR, locking the UI at certain moments to prevent the user dragging mid-search, providing back button support, and making sure we load our page resources in the most network-efficient way.

So, how did it feel as a programmer to move from VE to Google?  Overall, it was great; there were a bunch of times where it felt like Google’s Map API just worked better or was better organized than VE’s.  None of them were large issues, but they added up.  As examples:

  • Google provides both a movestart event and a dragstart event so that you can easily distinguish between map moves that are caused by programmatic calls to move the map versus map moves that are caused by the user.  Doing this in VE has often proved painful for us.
  • Fewer things happen in a timeout in Google Maps, which was a pattern that caused us considerable pain in VE.
  • Resizing seems like less heavyweight an operation in Google Maps, as the map doesn’t move by default when you resize.
  • Google seems more respectful of the global Javascript and CSS namespaces than VE.  VE, for example, adds methods to Object on some browsers, which breaks for-in loops.  They also define a style that has a selector of just “a img”.  What’s worse: that style has a !important on it, so overriding it is non-trivial.

There were of course a few issues with Google Maps (including at least one troubling bug we found in both maps.google.com and the maps API), but the overall experience was silky smooth.  And when we did come across problems, we found that, while the Google API docs were less thorough than comparable VE docs, there were many, many more forum and blog posts to help us find the answer.

Performance comparison adding 500 items to the mapAfter my weekend code-a-thon, I put the GMaps prototype up on a test server in our QA labs, and Dan ran a few of our standard performance tests against it.  The first thing he found was the map page consistently took 2.5 seconds faster to fire its onload event, seemingly due to the size and CDN-ness of the GMaps script.  Next, he ran our absolute worst case scenario test (driven by Selenium, naturally) where we add 500 houses to the map.  In IE6, we saw a speedup of about 10 seconds.  Even on a fairly fast browser like Firefox 3, the speedup was 3 seconds.  And that was in addition to the 2.5 second load speedup.

On the web, times like that are just gold; there’s just no way you can ignore a several second speed up.  Even though we had a lot of i’s to dot and t’s to cross, I knew at that moment we were going to make the transition.

My prototype seemed about half done and had taken two days to code; it took another two weeks of the search team’s time to get it up to production quality.  The difference between a product that was coded in a weekend and basically works and one that is coded and tested and works reliably for hundreds of thousands of people is massive.  As Glenn is always reminding us, 95% done is at best half done.

Anyway, hope you all like the end result, and I hope you find using Redfin that much snappier.  My thanks to all the folks who worked on the transition, especially Dan, Navtej, Jane, and Jim.  If you have any questions about the transition, feel free to ask in the comments.

* If you’re reading, Mom, no, it wasn’t literally meth-fueled.

Discussion

  • http://tr.ashcan.org/ Adam

    Nice job–congratulations! These stories are great and why I read the dev blog.

  • http://www.alexloddengaard.com Alex Loddengaard

    Great post, Sasha! Redfin is even faster now!

  • http://greg.abstrakt,ch Gregor J. Rothfuss

    Nice job! We love Selenium too. If you have any interesting API tests that we should be aware of, consider donating them to http://code.google.com/p/gmaps-api-issues/wiki/SeleniumTests

  • http://www.redfin.com Sasha Aickin

    @Adam and Alex: thanks!

    @Gregor: thanks so much for the offer; we may well take you up on it. We actually write our tests in Java for Selenium RC, though; is there any way for you to accept those type of tests?

  • Coventry

    Great job – love the post!

  • http://maps.google.com Gregor J. Rothfuss

    @Sasha: We picked Selenese since we thought it would be easier for developers to contribute, but there is no technical reason we couldn’t also accept RC tests. Send me an email and we can sort out the details.

  • http://blog.stchur.com Stephen Stchur

    To be brutally honest, the site actually seems quite a bit slower to me now, not to mention that the Google Maps tiles are not nearly as high quality (by which I mean, they look “cartoony”, whereas VE tiles look much more professional).

    Sorry, but I think this was a bad choice.

  • http://devblog.redfin.com/author/sasha.aickin Sasha

    @Stephen: thanks for the honesty. If you could let us know what specifically seems slower (first page load, dragging around with lots of houses on the map, zooming), how much slower it seems, what browser are you on, and where in the world are you accessing redfin from, that could help us diagnose the issue.

    As for the Google road tiles looking brighter and more cartoon-y, I agree, although I actually prefer it. Whenever I switched back and forth between VE and Google over the last few weeks, I found myself a little happier when I saw the Google tiles. However, I’m secretly a graphic novel fan, so that’s probably be a personal thing. It will be interesting to see how our userbase comes down on the graphical differences between VE & GMaps road tiles.

    Also, for what it’s worth, in our testing the Google satellite images tended to be higher res than the VE ones. Your mileage will probably vary depending on where you search, but hopefully it will compensate a little for your dislike of Google’s road tiles.

  • http://blog.stchur.com Stephen Stchur

    @Sasha:

    I’m in the Seattle area, so not far from you guys.

    I use (and was using) FF3, latest on Windows 2003 Server.

    The thing that struck me the most was panning the map. I pulled up the site (loaded quickly, I’ll give you that), and I punched in 98027 and got an initial set of pins/homes on the map. I then tried to pan just a little and it seemed to lag significantly. I felt like the map wasn’t responding to my mouse movements. Perhaps this is due more than anything to the refresh that is trying to happen while I pan?

    Not sure, but it was just an initial impression that the panning felt somehow slower.

    I guess the tile “style” is very much personal preference, as you said. I felt like the VE tiles looked more like a “real” map, but I’m sure I’d probably get used to Google ones if I used Google Maps as my main maps & directions app.

  • http://blog.caffeinatedsoftware.com Robbie

    Hey guys, several points…

    1. I’m impressed that you were able to get the ParcelStream tile layer to work given the vastly different query string format MS & Google have for the map tiles. Google has x,y,z parameters in it’s tile query strings & MS usually has one number to represent all 3 parameters in it’s tile URLs. It’s cool you figured out the conversion with seemingly little trouble.

    2. Google Street view is slick isn’t it? MS needs to get a street view implemented now that Google has nearly half the country in street view. I’ve been visiting downtown Seattle a lot recently, and found it very useful (& also very buggy, but useful).

    3. My impressions are Google map tiles always download faster than Virtual Earth (never understand why because I’d assume Google & MS Seattle-area data centers are both than less than 10ms away from my PC so it shouldn’t be a Google CDN advantage).

    The map control initializes faster w/ Google than VE (haven’t measured download time), it just seems that way. As for push pins, I haven’t noticed a real major difference between them (both seem fast). I wish you would add mouse wheel support though.

  • http://www.redfin.com Sasha Aickin

    @Stephen: it seems all right for me, and faster subjectively than it was in VE, but you have indeed found our worst case performance scenario. In VE, we heavily optimized the scenario where you drag the map and only one or two pushpins change. In Google, we’re just redrawing everything every time. In the very worst case (499 houses stay the same and 1 changes), we saw slight performance decreases with Google (on the order of 10%/100ms). Google won in every other scenario we ran; once we started changing a few dozen houses or putting less than 300 or so houses on the map, Google was the clear winner.

    We thought about implementing some of the same optimizations we had done in the VE world for this small change case, but in the end we decided that we didn’t have time in the schedule for the potential destabilizations and that we could take that slight perf hit in the one case to get the perf gains we saw in every other test. We may well devote some time to re-implementing some of the optimizations in Google in future releases.

    @Robbie:
    (1) I’d love to take credit for this, but ParcelStream just provides a different API and tile server for Google. It was not due to any magic on our part.

    (2) We actually had StreetView integrated before this switchover, and I do really like it. Sometimes it doesn’t tell you much more about the house, but when it does, it tells you a lot. I’ve seen listings with one or two artfully taken internal photos and then a really horrible StreetView.

    (3) I’m 99% certain that Virtual Earth is still serving everything out of a data center in the eastern US, not out of Seattle, which means that the tile loading is almost certainly a Google CDN advantage.

    As for pushpin adding, I don’t think you see a major difference until you add a ton of them. If you add 10 or 20, the difference will be in the 10s to 100s of milliseconds range. When you add 500 pushpin/markers to a map, it can be several seconds, and it’s highly dependent on the map platform and the technique you use to do the adding.

    Finally (this comment is nearly a full blog post in itself), we had mouse wheel support a while back with VE. After watching users in testing completely flail and not understand why the map was jumping around when they accidentally nudged the scroll wheel, though, we disabled it. Sorry.

    As a favor, though, if you drag this link to your bookmark bar, you should be able to click it on the Redfin map page and enable scroll wheel zooming:

    Enable scroll wheel

    You’ll have to click it each time you go to the map page to enable, and I make no guarantees that we won’t break it in later versions of the app. For the time being, though, enjoy.

  • http://devblog.redfin.com/author/sasha.aickin Sasha

    Argh. The blog software mangled the URL for that Javascript bookmarklet. Instead, create a bookmark, and make its address:

    javascript:window.g_searchController.map.map.enableScrollWheelZoom();

    That should enable scroll wheel on the map page.

  • http://www.earthware.co.uk/blog brian

    Id be interested in seeing how you speeded up googles addoverlay method as all my tests show google maps is 4 times slower for adding 500 pushpins that virtual earths bulk add method?

    Brian

  • http://devblog.redfin.com/author/sasha.aickin Sasha

    @brian: For right now, we’re keeping our speed enhancements to ourselves, although we have talked internally about open sourcing the solution. Keep your eyes on the devblog, where we’ll surely announce if we change how we treat that code, and sorry I can’t be more helpful right now.

  • david

    I see brian already asked the question that I was going to ask in this thread. I was wondering if you guys are actually adding up to 500 pushpins to a map or if you are creating image layers of the pushpins (and cacheing them) at zoom levels that result in a large number of pushpins being in view? Can you really add 500 pushpins to a map and expect decent performance?

  • http://www.Vishcio.us Vish

    Hi Sasha,

    I would be interested to read about how y’all went about creating a custom GOverlay.

    Thank You,
    Vish

  • http://www.pictometry.com Jeff Marcus

    Pictometry “Birdseye” imagery is available to those organizations who wish to utilize any mapping program including Google.

    The API is called Navigator API and includes many features such as more dynamic viewing, measuring, annotating and even knowing the date!

  • http://www.redfin.com Sasha Aickin

    @Vish & @david: as I replied to brian, we’re not quite willing to talk about the internals of the solution yet. I wouldn’t be surprised if we do soon, though, so keep your eyes on this blog.

  • mike ridgeway

    Seriously – what good is speed if it comes with the attached lack of “completeness”. Great – you get us the maps 2x as fast, but half of the map is missing. What good is that? There has to be an economic rationale behind this decision. no?

  • http://devblog.redfin.com/author/sasha.aickin Sasha

    @mike ridgeway: What do you mean by “half of the map is missing”? If you’re seeing errors, please send us a screenshot of what you’re seeing to techsupport@redfin.com and we’ll try and figure out what’s going on. Thanks!

  • Grank

    This post is sorta frustrating for me.
    I’m used to working with VE and being able to add large numbers of points without performance difficulty. My latest client made GMaps a requirement instead, and using the out of the box google API, I literally cannot accomplish what the client has requested due to the poor performance with large numbers of points. I’m already clustering pins into groups when zoomed out to improve performance, but that doesn’t help as we zoom in; unlike VE, GMaps performance seems like it’s degrading with many points on the map at the same rate regardless of whether or not they’re visible in the map’s bounds. The last couple of zoom levels become almost unusable.
    I was surprised by that, and sought more information; this post came up in my search for performance comparisons between the two mapping systems…
    I think it’s a little disappointing to acknowledge that the Google Maps API has major performance issues with lots of stuff on the map when compared to Virtual Earth, and then turn around and say “no problem! We fixed it and now Google Maps is several seconds faster than Virtual Earth”… and put up performance graphs lauding the increase over VE and everything…
    …but do it by having to code your own custom GOverlay, effectively re-writing yourselves the functionality that was slow in the original API.
    And then not post any code or any implementation details or clues.
    It’s good to know that it can be fixed and that people out there have fixed it, but I hope that Redfin will consider releasing the source code soon, or at least discussing some of the details so that someone else could reproduce something similar for the community.
    In the meantime, I guess I’ll have to try to hack around it myself. Maybe that in-bounds thing I mentioned above is where I should start? Or can you give us any other clue?

  • Angus Davis

    I read this post with interest a few weeks ago. Then today, I stumbled across this blog post:
    http://viswaug.wordpress.com/2008/12/18/creating-a-lean-and-fast-custom-marker-overlay-for-google-maps/

    … in which a guy named Vish discusses a method he learned about to solve this problem from Pamela Fox. I wonder if Redfin used this approach, or something different?

    Vish says: “The sample below illustrates how to create a light weight custom marker GOverlay for Google maps that is REALLY FAST. And it allows for a multitude of customizations as you see fit. … The demo below also allows you to check out the performance of the custom overlay with large number of markers loaded.”

    Anyway, since several folks in these comments complained about not hearing how redfin solved the problem, I thought it would at least be worth pointing to this other fellow. I have never built anything in Google Maps so I don’t know much, but I do know that no company has a monopoly on smart people. Maybe this Vish guy is onto something.

  • Bonnie

    This is a really great post! I stumbled on it trying to solve the problem of a low-cost DESKTOP api for mobile computer systems (in-vehicle).

    What I find interesting are the parcel polygons from ParcelStream. Although the overlay intermittently fails to completely draw on zoom, a quick refresh does the trick, and the data is very attractive!

    I did a POC converting shapefiles in ArcGIS to kml to show existing GIS data can be displayed on a Google map. It worked, but I was really dissatisfied with the graphic quality, especially when zooming in; the lines dissolved into gigantic pixel blobs. Coarse and choppy would be an understatement.

    I am curious; What is the source of the city boundaries? TIA!

  • Treapletroulp

    Im trying to locate a pair of H.I.D bulbs for my wife’s 1993 Acura and was wondering if anyone had any guidance?

    We have researched and read on these websites for specifications but any first hand information would be really helpful.

    hid light

  • http://onlineslotmachines888.wordpress.com/ anneterob

    Hi, cool site, good writing ;)

  • Pingback: Rwanda « Honeysun

  • Rob

    y is google maps & earth so out of date

  • http://blog.caffeinatedsoftware.com Robbie
  • http://www.hidonlinestore.com/ HID Light

    Great article write now.Thanks for information on topic.

  • http://www.discount-nike-dunk-shoes.com nike dunk shoes

    Hhe article's content rich variety which make us move for our mood after reading this article. surprise, here you will find what you want! Recently, I found some wedsites which commodity is colorful of fashion.
    http://www.scarf8.net

  • Jimmy

    Thanks sasha, now I know what to do.