October 22, 2007
Listing alerts is a Redfin feature that lets you keep track of new listings in your area. You start by doing a search on our map page, and then you click on the “Save this Search” link:
This lets you assign a name to your search and save it. You can also sign up to get a daily email with new listings that match your criteria.
We recently released an upgrade to the Redfin website. There were a few minor tweaks to the listing alerts code, but nothing significant. Still, a few days later we got an email from a customer saying (to paraphrase): I used to get an email alert almost every day, but recently they stopped, what’s up?
Being the owner of the Listing Alerts feature, my heart skipped a couple of beats. Then I jumped in to analyze the problem. What I noticed was that indeed this customer (Norm) had a steady stream of emails that mysteriously ended. However, the emails didn’t stop on the day of the release; they stopped several days later. I also noticed that Norm made changes to his saved searches right around the time the emails stopped. So, my hypothesis was that Norm’s new searches were somehow more restrictive and we simply didn’t have results for them. To verify this, I did a test run using two of Norm’s searches, an old search and a new search, for the last 24 hours of data. I found that the old search returned results and the new one didn’t, which seemed to confirm my suspicion. So I went ahead and replied to Norm saying: please relax your criteria and try again. Luckily for me, Norm is a stubborn guy. He replied a few minutes later saying: actually, I think my new search is more broad than the previous one.
Now I was really worried. I took another look and I found the smoking gun: for all saved searches created since our recent release, the search area parameter was not saved correctly. To explain what was wrong about it, I need to give some background on how we store your saved search…
Our saved_searches table was changed a little in the recent release. Prior to the release, it looked like this:
[saved_search_id] [name] [min_latitude] [max_latitude] [min_longitude] [max_longitude] [other criteria…]
After the release, it looked like this:
[saved_search_id] [name] [polygon] [other criteria…]
Why did we make this change? Well, this release was also Redfin’s “toe dip” towards changing our database platform from MySQL to Postgres. We found that Postgres gives us better performance on certain kinds of searches (e.g. past-sales searches). So, in this release we replicated our past sales data from MySQL to Postgres and changed the website to issue all past-sales queries to Postgres. Now, to get the performance benefits from Postgres, you have to store your regions in a column of type GEOMETRY, as opposed to simple columns of type DOUBLE or whatnot. So, we wrote the code to store and fetch GEOMETRY types. And once we had it, we figured we may as well use it for saved searches as well — the search area is just a POLYGON after all.
So where’s the problem? The problem is that while in MySQL coordinates are typically stored as (latitude, longitude), Postgres prefers to store them as (longitude, latitude). Or, to be more specific, Postgres doesn’t particularly care which order you use, but the Java library we used (Vivid Solutions) opts for the (longitude, latitude) style. Why does it do that? Probably because it thinks about these coordinates as points in X-Y space, where X is like the longitude (east/west = right/left), and Y is the latitude (north/south = up/down).
We actually noticed this discrepancy when we were close to our code-complete milestone, and we had to make a decision about it. One option was to standardize — to make all coordinates in both databases follow the same (longitude, latitude) style, the style we will eventually adopt when we move to Postgres 100%. Another option was to keep keep MySQL coordinates (latitude, longitude) and to only use (longitude, latitude) in Postgres. We chose the latter approach because it was less disruptive and we were running out of time. Only we didn’t get it quite right — we missed the code used for the saved searches table. The effect was that all existing saved searches kept their (latitude, longitude) format and continued to work, but all new searches were saved as (longitude, latitude) and were busted.
How did we miss this in our testing? I will be the first to admit that we don’t do as much testing at Redfin as we could or should. However, we did have about 40 unit tests and 70 integration tests, and they definitely included scenarios that checked listings that are in/out of the search area. The trouble was that our test cases were too “symmetrical”. This is best explained by an illustration:
Well, when we discovered this problem, we spent the first few minutes being horribly embarrassed for letting it slip through. Then we buckled down, found a solution, and released it later that day. We also added a bunch more test cases to catch the problem in the future. And we sent an email to all the affected users with our apologies for the missed emails. At this point we believe listing alerts work again.
So, what are the lessons?
Strive for consistency in your data, even if it’s more painful and/or takes more time.
Avoid dual-DB situations. Both MySQL and Postgres support the SQL standard, but there are enough differences between them to make life difficult, and this lat/long issue is just one example. Now, we chose to take a toe-dip, and if we had to do it over again I think I would have made the same decision. In other words, do as I say, not as I do. 
Pick asymmetric test cases. Discussed above.
Add constraints. Our database didn’t complain when the coordinate (47.5, -122), which is in Seattle, was rewritten as (-122, 47). Take a moment to think about that. A longitude of 47 is probably somewhere around India. A latitude of -122 doesn’t exist: it’s beyond the south pole! Why didn’t the database complain? Because as far as the database is concerned, this is just a point, and it’s a perfectly valid point. If you can afford the performance hit, it helps to add constraints to tell the database: in *my* application, a Y value of -122 is no good.
Don’t repeat yourself. The search area for a saved search was actually stored in two formats for two purposes. The first purpose was for evaluating the saved searches for the purpose of sending emails, as discussed above. The second purpose for interactive use – where you can simply pull up your saved search on the website and click “run search”. In the second case, we chose not to use the bounding box because your view may be a little different each time you pull it up, depending on the size of your screen or whether the window is maximized. For interactive use, it’s better to use the center point of your search and a zoom level – this combination can adjust more gracefully to different situations. The problem in our case was that we separately stored the latitude and longitude for interactive use. Had we simply derived them from the polygon’s center, we would have caught this error earlier.
Image credits: http://www.slashfilm.com/2007/02/20/movies-that-should-have-won-an-academy-award-but-didnt/
October 22, 2007
Last week, Microsoft released a beta version of a tool called MapCruncher. MapCruncher allows you to take a rendered image (GIF, PNG, JPG) and create a set of map tile layers that can be then drawn over a Virtual Earth (VE) map with the correct projection.
Since Redfin’s map is based on VE’s platform, I spent a couple hours using MapCruncher on a test project to see what we could potentially do with it. My project was to take an image of the BART routes in the Bay Area and place it on the map.
It worked out pretty well. A screenshot of the resulting mashup map that was created, or you can just check out the resulting mashup yourself.
What did I learn about MapCruncher?
- It’s really easy to use. It provides a side-by-side view of your desired overlay image and a live VE map view. You simply line up landmarks in the cross hairs of each view and mark it as the same location.
- It’s pretty quick. I had generated the first version in over an hour and it worked.
- Your overlay images need to be very basic and have no text. The original route map image I tried had all the location names and some other features drawn on it. When it was re-projected, the text was completely warped out of shape and unreadable. I used a image editing tool to remove all the text, background colors, etc. - everything except the routes themselves. That simplified route map image considerably cleaned up the resulting mashup.
- More landmarks will definitely improve your results. This may seem obvious, but MapCruncher says it only needs around 10 waypoints to render the mashup. I set 10 along the west edge of the map and two on the right. It did great alignment on that left set of points, but the projection was way out of whack along the right in Oakland and Pleasanton. I ended up setting 21 points with an even mix across the different quadrants of the map.
- Zoom levels will make you want two source images. I set the render to only go to zoom level 11 first, but didn’t like how soon the route map disappeared when I zoomed in lower. So I cranked it up to render all the way down to 13. Since it’s all doing it from one image, the routes were drawn gigantically at lower zoom levels. I’m guessing you probably need to have different source images for different sets of zoom levels to account for size differences.
Here’s the components of my project if you want to try it out yourself:
MapCruncher seems like its great for either a simple image overlay or a quick and dirty project. If all we had was a rendered image of some information, I could defintely see it as a way to get that data integrated. Honestly, I can’t now imagine trying to generate a set of projected tile layers of it without MapCruncher.
You’ll probably have to spend a fair amount of time getting the overlay images just right so that they work at all zoom levels. You’ll probably want to play with transparency as well to make sure it doesn’t completely obscure the map. At some point, I’ll have to try rendering the image from raw GIS data in a tool like ESRI to see how that improves the results.
You’re probably wondering when this will show up on Redfin’s map. I’m not sure it will. We already display all the BART stops on the map thanks to VE. Adding the route overlays actually adds a lot of clutter to the map without providing a lot more information.
We could certainly be wrong on that though. Drop us a comment if you believe this is very valuable info to add to the Bay Area map or if you have suggestions on what other data overlays we should add to help in your house search.
October 9, 2007
One of the main goals of our latest release was to improve the overall performance of the user interface, so we’re starting an intermittent series today on the dev blog talking about what we learned along the way.
Firstly, we had to recognize (channeling Steve Souders of YSlow and High Performance Web Sites fame) that the largest part of our performance problems were on the client-side, and the problems were most severe in IE6.
We set about trying to optimize client performance independent of network latencies and server query times. For the most part, this meant reducing the time the browser spent running our JavaScript. One reason this is particularly important is that the browser does not do anything else while JavaScript is running; the UI is completely locked up. No events, no back button, no browser menus.
If you do a lot of heavy processing on the client side in JavaScript, this can be a real problem since it causes visible delays and makes a web application generally clunky and unresponsive. Accepting that there were times when our website was sluggish, we set out to improve the overall performance of our user interface. More about how we figured out where to start after the jump. Read the rest of this entry »
October 8, 2007
I arrived at work today with a perplexing tech support e-mail in my inbox. A customer was claiming that although Redfin had never worked well in Firefox, last week we totally broke it. Needless to say, we were all a little surprised (and more than a little worried) over here, since we use Firefox all day every day. When we asked the user to send us screenshots, they looked something like this:
This is when we got well and truly confused. It looked like the browser was downloading our CSS, because items were appearing in the right place on the page, but all color had drained out of the world. Was Firefox in some sort of a funk? At first, we guessed that perhaps the user had some sort of ad blocker that was blocking images from our content delivery network, but we quickly noticed that some images made it through. What was going on here? Solution after the jump.
Read the rest of this entry »
October 1, 2007
UPDATED: As of January 31, 2008, Redfin supports Safari 3 without any of the trickery described below. We’ll leave the post here for the curious, but in the meantime, there’s no reason to go through these steps on our account.
Last January, we ported Redfin from a proprietary map platform to Microsoft’s Virtual Earth, a move that has enabled us to move faster in adding new markets and features. We’ve gotten both praise and criticism for the move over the last 9 months, but no piece of feedback has been as constant or as vociferous as the cry of Mac users who want us to support Safari.
As a MacBook Pro user myself, I understand the desire for us to support the native Mac browser (of course, we do support Firefox on OS X), but Virtual Earth didn’t support Safari, so we were pretty much out of luck. And although I like Safari, we have to admit that Safari 2 is buggy at best with support for CSS and JavaScript. Applications like Redfin or Virtual Earth that use a lot of client-side scripting are bound to have problems with Safari 2. This doesn’t mean that it’s impossible to support Safari 2, but it is likely to require significant testing and engineering to get a site to work properly.
The upcoming Safari 3, though, has taken a massive leap forward in terms of standards support, and it’s crazy fast to boot. While we don’t support Safari yet officially, you can use Safari 3 with Redfin if you trick Redfin into thinking that you are running Firefox, and it Mostly Works ™.
Before you start, please note that Safari 3 is in beta, it is not to be used on mission-critical systems, this is all at your own risk, your computer could spontaneously explode, etc. With that disclaimer out of the way, here’s how you get Safari and Redfin to play nice:
- Download a copy of Safari 3, either the Safari 3 Public Beta from Apple or the WebKit nightly from webkit.org. The Safari Public Beta is a better tested build, but it may not have all the bug fixes of the nightly build from webkit.org. Also note that installing the Safari 3 Public Beta will get rid of Safari 2 on OS X, whereas I believe (but am not 100% sure) that the WebKit nightly can co-exist with Safari 2.
- Turn on debug functionality for Safari. On Macs, do it this way:
- Quit Safari so it’s not running.
- Run the program called “Terminal” which is located in your Applications->Utilities folder.
- Type the following command into Terminal exactly as shown:
defaults write com.apple.Safari IncludeDebugMenu 1
- Quit from the Terminal program, and launch Safari again. Safari should now have a “Debug” menu. If not, check the command you typed above. Case is important.
On Windows boxes, turn on the debug menu this way:
- Quit Safari so it’s not running.
- Open the file at:
c:Documents and Settingsyour usernameApplication Data
Apple ComputerSafariPreferences.plist
- Add the following text to the file before the final </dict>:
<key>IncludeDebugMenu</key>
<true/>
- Launch Safari again. Safari should now have a “Debug” menu.
- Launch Safari 3. In the Debug menu, select User-Agent –> Firefox 2.0.x.x. (The exact version of Firefox in the Debug menu depends on which build of Safari you have.)
- Go to www.redfin.com, and use Redfin to your heart’s content.
There are a few things to note here. First, you will have to switch the User Agent every time you open a new tab or restart Safari. Second, there are also a few known bugs we’ve seen with just the limited testing we’ve done:
- Depending on which version of Safari 3 you get, the images of houses next to the map may be stretched weirdly.
- The zoom bar on the map isn’t centered correctly.
- Keyboard shortcuts on the map page (up/down/left/right) don’t work correctly.
If you try this out and find any other bugs (and we’re sure they’re there), let us know in the comments section.
(Photo credit: p_c_w on Flickr)
