This time around, Jamie DeMichele, our Technical Lead, decided to take on adding the ability to search for listings by school attendance areas as his third-quarter objective in 2012.

What’s the big deal? It turns out that schools play a crucial role in the real estate industry, both in terms of helping buyers decide where to live, as well affecting resale values for homeowners. From the perspective of a Product Manager on the team, it is simply a dream come true when one of these side projects happen to be one of the most requested features by our visitors as well as real estate agents.

In Jamie’s own words, “after working on mostly backend functionality for many years, getting the chance to work end-to-end on a feature was an awesome experience!”

“Adding search functionality via school boundary polygons is a huge benefit to our customers as well as our agents, especially when it didn’t require massive upgrades to our existing infrastructure. We also built an easy way to regularly update our school boundaries, ensuring a high quality bar for school boundary data on Redfin.”

As usual, a big THANK YOU goes out to everyone on the engineering team who contributed to this feature. We truly enjoyed building a new way for our customers to search for homes, and we’re thrilled to continue reinventing real estate in the consumer’s favor.

This was our world as of last year. We had home brewed approaches to gather basic performance information. We knew which pages saw the most use, and we knew if they were getting slower or faster. However, we didn’t know why our pages were slower or faster. We could improve our performance, but it was an involved process.

Enter New Relic!

About this time we started looking into New Relic. New Relic monitors application performance. Their code is easy to set up, and reports on a number of metrics. Some of our favorites were the slowest requests on the servers, the slowest database queries (and tables), traces of particularly slow web requests, and the aggregate errors being reported across our servers. There are plenty more features, but these were our favorites.

The overview of our site’s performance in New Relic

Initially, we had a couple reservations about using New Relic. Mainly we were concerned with how it would affect our website performance. After all, New Relic adds an additional wrapper around our code, and it is sending performance data from our server to the New Relic server. We did some investigations and we found that the performance hit of using New Relic should be less than a 5% hit to our memory usage and CPU usage. In addition, the data passed to New Relic is sent asynchronously. If the New Relic server is down, then the local New Relic monitor will store data for up to 15 minutes, and then start dropping data. With these answers in hand we were willing to take the New Relic app for a spin.

New Relic FTW!

New Relic won us over when it helped us catch an error we were not even aware of. We have some assistant servers that pre-render some of our pages, in some cases these assistant servers were throwing an error when they tried to write out a response. However, these errors were being caught (incorrectly) before we could log them. According to our logs, there were no errors. However, New Relic kept recording a disturbingly large number of errors on these servers. We investigated and fixed the root cause, but it was a tricky error. The fix could have taken a lot more time if we hadn’t know the exact urls that were causing the errors and seen the stack trace of the issue.

The New Relic error that helped us track down our silent server error

Is New Relic the Silver Bullet™?

New Relic is an extremely useful tool, but at the end of the day it’s a tool. What you get out of it depends often depends on how well you understand the inner workings of your application. For example, we have a database table, logins, which sees a lot of use. If our database hits high loads then logins tends to take up a lot of time. So when the database backs up it tends to look like the logins table is holding everything back. However, the slow login table queries are merely a symptom of a high database load and not the cause.

Misleading Login spike during DB slowdown

In addition, New Relic is built as a website monitoring tool. It has the ability to monitor jobs, but it takes some digging to find out how to configure the New Relic agent for a job. When we set up jobs we had to change the newrelic.yml file to have the following lines (these changes may be Java specific, and there must be exactly two spaces to the left of each line in the config):

  sync_startup: true

  send_data_on_exit: true

  send_data_on_exit_delay: 20

Stack traces on jobs also leave something to be desired. New Relic only traces the first ~2000 method calls. This is fine in a web transaction, but this will only scratch the surface of long running jobs. Nevertheless, it’s nice to get some insight into how our jobs are influencing our database loads.

Final Thoughts

New Relic helped us dive into our code in greater detail than ever before. Using their application, we could find out not only which pages were slow, but crucially we could find out precisely which queries and functions were slow. Now it is easier than ever to hunt down performance issues, which has led to an unanticipated benefit. Prior to New Relic, we had one dev who took on the mantle of performance guru. He let everyone know when pages were slowing down, and encouraged the teams in charge of those pages to fix them up. Some days it seemed he had a Sisyphean task. He could harp on the performance of a page, and it would slowly improve, but in the meantime other pages would slow down. Now that New Relic is in place there has been a noticeable shift. Teams have started to jump in and fix their own pages spontaneously. Even teams that don’t own code have been helping improve database performance.

We have had a great experience with New Relic. If you want to give it a spin, check out www.newrelic.com!

Note: the screenshots in this post have been slightly altered to exclude some internal information.

If you’re a frequent visitor to our site, you may have noticed something different about the page today. After years of living with a blue color scheme we affectionately called “cadaver blue”, we’ve finally gone and put the “red” into Redfin. The new look is just the latest — and most visible — example of our efforts to strengthen and evolve our brand. For example: remember that tree in our logo? Yeah, me neither.

A Look Into the Kitchen

Redesigning an existing site can be very complex. Even pulling off a relatively small-scope change such as updating our blog theme can be so daunting that many companies and their designers decide to start from scratch. Why? Because when you’re updating an existing site, the hard work tends to be less in the design process and more in the updating for an ancient code base or normalizing different ways an element — like a button — can be put together.

To put it another way, which do you think is easier: cooking a great meal from scratch, or taking a bunch of leftovers and making a meal that tastes better than the original?

Back when the internet was younger and the technology was simpler, there were more complete redesigns done, but as the net has matured and the number of people relying on sites has grown, “redesign” has given way to “refine” and “iterate”. That means more incremental changes — page by page, or section by section. That’s the approach we took.

But It’s Not Just a Fresh Coat of Paint

When we started the effort to update our web site’s visual branding, we took a long, hard look at the common elements that make up our site – the “widgets”, if you will. We’re in the midst of unifying and updating them on our site, though it’s a long term project, and our refreshed home details page was just the beginning.

Redfin was built by many hands over many years, using different programming languages, styles, and coding practices. Eventually we’re going to unwind all this and have everyone using the same set of widgets, but in the meantime, we’re using the power of CSS to bring some visual similarity to the elements that have the same purpose — page navigation, buttons, tooltips, dialogs, etc. One of our goals is to simplify the elements you interact with, just like we want to simplify real estate.

It may be a lot of small changes, but in this case, small changes make a big impact. Just by removing the blue backgrounds we reduced the “boxiness” — the heavy framing of some elements — by an astounding 70%! Okay, so maybe that’s a made-up metric, but it still illustrates the dramatic effect we were able to achieve. One small change that gives the site a night-and-day kind of difference.

Express Yourself

We’re also starting to express Redfin’s personality with our new color palette — bright, vibrant, engaging, alive. That’s how we want people to see Redfin, because that’s just what we are as a company. Compare that to cold & confined — two words we heard about our old theme — and you feel the disconnect in who we are and how we presented ourselves to the world. Now our outsides have started to match our insides.

These changes have been a long time in coming, and by no means are we done yet. In the past year, we’ve doubled the size of our design team between our Seattle and San Francisco, which means these changes are going to be coming fast and furious as we work on improving the look of other elements and sections of the site, which you’ll be seeing in the coming weeks and months.

So a huge thank you to everyone who has helped make this a reality over the past few months. This was a labor of love, and I was proud to have played a part in its creation. And to our dear customers, we hope you will enjoy our face lift. Let us know what you think in the comments below.

That said, we’re thrilled to finally be putting our fresh, new face forward.

After a very successful night of hosting the Seattle Scalability Meetup few weeks back, we decided to host another one! This time, we hosted the Seattle JS Meetup. Just like last time, we had an incredible turnout from the Seattle community, with more than 100(!!) attendees sitting in our newly renovated office.

This time, we started off with lightning rounds, where anyone can come up and give a quick, two minute presentation about anything JavaScript related. After the lightning rounds, we had our first speaker, Yuriy Dybskiy, who talked about Meteor, a JS framework that allows developers to quickly create and deploy a web application. He went through and showed us how to create a simple JS web app that allows users to log in and show a picture of their avatar (if they have Gravatar account associated with the email), all using JavaScript and the Meteor API.

Next we had Ryan Eastridge, who created Glowstick, a cool API that allows you to control LEDs on a custom made Arduino board. He created this in part to introduce kids and young adults to the world of computer science and programming. Using simple syntax and APIs, kids can go from creating simple light patterns to making intricate moving rainbow lights (note: cat not included).

Ryan actually brought in two custom-made light boards to the meetup for the attendees to try. The board was connect to our local network and anyone could access it through the network. All they had to do was open up a web browser and navigate to the board. Next, open the console on the browser, and start programming LED light patterns. During the presentation someone actually made a semi-working game of Pong using the Glowstick API. Redfin engineers also created a Redfin Logo using the Glowstick API.

After the talks people mingled while some continued to program cool lights into the light board. Overall it was a great experience meeting other developers in the Seattle area and hearing all the cool projects people are working on.

We’ll be hosting more awesome events in the future. Until next time!

Elements Tab

To access Google Chrome developer tools, right click on the element on the page that you want to see the HTML of (or anywhere if you just want to open the tools) and select Inspect element. This will take you to the Elements tab. Here you can see all of the HTML for a page and all CSS styles that apply to each element. Here are some uses for the Elements tab:

• Edit HTML directly in this view by right clicking on the element you want to change and selecting Edit as HTML.
• Notice how all of the styles that apply to the selected element are in the right sidebar of this tab.
• Unselect styles to make them no longer apply by hovering over the style you don’t want and deselecting the checkbox next to it, or you can edit the styles by double clicking on the style you want to change.
• If the style is crossed out, then another style has overwritten the value of that style, if the style has a yellow exclamation mark next to it then the browser doesn’t understand what it means.
• Create new CSS selectors in the dev tools directly by clicking on the little plus sign above the Style’s section, this can be useful if you’re trying to figure out how to select a certain element.

Elements tab of Google Chrome developer tools

Sources Tab

Chrome’s javascript debugger can be found in the Sources tab of the developer tools. Here are some of the uses of the Sources tab:

• (This is bold, because it’s important!) Before you start debugging, you should set the debugger tools to pause when there is an uncaught exception. To do this, press the stop sign in the bottom left corner with a pause symbol in the middle of it until it turns purple, then scripts will pause when there is an uncaught exception and you’ll be able to see what is causing the exception by checking the values of local variables at the point where the exception is thrown. You can also set it to pause on all exceptions, but this can be very noisy, and usually if there is a caught exception you were already aware of it.
• To use the debugger, click on the arrow in the top left corner to see all of your javascript files. Browse through the scripts until you find the one you want to debug.
• To set a breakpoint, click on the line number to the left of the line you want to set the breakpoint at. Once the breakpoint has been hit, there are basic stepping and running buttons in the top right corner. Another way of setting a breakpoint is to put “debugger;” directly in your code, the Chrome developer tools will recognize this as well.
• When you are paused in the debugger, look at the right sidebar of this tab. You can see all local and global variables and their values in the Scope Variables section.
• You can see the stack trace in the Call Stack section, and you can see all breakpoints that you have set in the Breakpoints section. The breakpoints will persist on page reload.
• Edit javascript on the spot in the developer tools, just edit the file in the Sources tab, then Ctrl+s and the new javascript will run, but this does not persist when the page reloads. I’ve never used this functionality, mostly because you can’t make the js persist, and most of the time you want it to.

Sources tab of Google Chrome developer tools

Console Tab

The Console tab of chrome developer tools has a lot of useful features. The console is also accessible from other tabs by clicking the Show console button in the lower left corner. Here are some things you can do with the console tab:

• To show information about the code in the console, use console.log(string) in your code, this is really useful for logging errors, or seeing how long something took to run.
• Right click in the console and select Log XMLHttpRequests Then you can see whether your XHR requests have been successful or not directly in the console. You can also see the details of these calls in the Network tab.
• Type the name of a variable in the console and see it’s value. You can dereference the variables you find in the console to find their fields. The console will autocomplete the variables for you showing you which variables that it knows about in the current scope.
• Query for HTML elements in the console using jquery/dojo/whatever you use. Ex: dojo.byId(“my_object_1″); Or select the element you want to see in the Elements tab and type $0 into the console.
• You can also execute javascript in the console, it will execute in the current scope, so if you’re paused in the debugger you can set variable values to something else in the console and see how it behaves.
• Use the monitorEvents method, pass it an element and you’ll see all events that are being fired on that element.

Debugging in Internet Explorer

If you have a bug specific to Internet Explorer (which you probably will at one point or another), you may need to use their debugger tools. They are a little harder to use than Chrome’s tools. I do most of my internet explorer debugging by googling my problem, a lot of people have problems with IE, so try that before learning how to use their tools, I get by mostly without them.

To open Internet Explorer’s F12 developer tools, press F12. F12 developer tools allow you to change the Browser/Document mode of internet explorer so that you can see how it looks in previous versions of Internet Explorer, to do this click on Browser Mode or Document Mode and select the version you want to see. However, beware of relying totally on testing in IE9 set to IE8 mode, we once had a bug where we tested everything using the developer tools in IE9 and setting the browser and document mode to IE8, but if you were actually in IE8, you would get a really ugly error popup that said “Stack overflow”, we didn’t find this until after deploying, so I recommend you also do some testing in real IE8 before deploying anything big. To inspect HTML go to the HTML tab, press the little refresh button, then press the pointer button and click on the HTML element on the page that you want to see. CSS and HTML will be visible here. The HTML in the tab only updates once you press refresh so if you click something to change the state, you have to refresh the developer tools again.

HTML tab of Internet Explorer F12 Developer Tools

To debug javascript, go to the Script tab, press the Start debugging button, then you’ll see one really long file with your scripts in it. The best way I’ve found to navigate this is to copy the line of javascript that you want to set your breakpoint at, and search for it in the Script tab. You should be able to find the line this way. Then you set breakpoints and step through the code with the basic debugging buttons in that tab.
Script tab of Internet Explorer F12 Developer Tools

Redfin Engineering had an incredible time hosting the Seattle Scalability Meetup earlier this week. In the past, we’ve had celebrity guest speakers give talks in our San Francisco office, but we’ve just never been able to make it happen in Seattle before. (If you’re curious, see Ryan Dahl on node.js, Alex Payne on how and why Twitter uses Scala, and Jeff Hammerbacher on Hadoop.)

Part of what made this possible now is the fact that we recently expanded and remodeled our Seattle headquarters.  Our new space includes a meeting room overlooking the waterfront that was able to accommodate the 100 attendees for this vibrant meetup.

We were treated to two excellent technical talks.  The first was from Paul Gross at Braintree, who spoke about how Braintree supports high availability.  The talk covered how Braintree has engineered every layer of their stack to be tolerant of both planned and unplanned outages.  Braintree has open sourced quite a bit of the code they use to accomplish this, including a custom load balancer and scripts for handling automated failover for Postgres.

Next, we heard from Mike Miller, the Chief Scientist at Cloudant.  Cloudant describes its product as “a globally distributed, document database-as-a-service that scales from megabytes to petabytes.”  Supporting geospatial indexing and querying is a key part of Cloudant’s value-add, so their team has taken a hard look at geospatial in technologies like PostGIS, and then applied those learnings to features that work on their globally distributed database.

Afterwards, we met up for a beer at Sonya’s, which is just down the street from Redfin HQ.  We’re looking forward to hosting this Meetup again in April.  We’ll also be hosting the Seattle JS meetup for the first time on March 14th (http://www.meetup.com/seattlejs/events/104167682/).

Hope to see you there!

At Redfin, the Marketing team contacts customers all over the country about upcoming classes and events near them. We don’t send a lot of email to each recipient – nobody likes spam – but we do send a great deal of small campaigns each week, and they add up: this year, Redfin will put on almost 1,100 events, and year to date we’ve sent over 2.8 million emails promoting them. There’s so much going on that early this year, the team hired a third coordinator to further distribute the work involved in planning, scheduling, and promoting all these events. At the time, the process for building out an email for a class took somewhere under an hour, two or three different pieces of software, and a lot of manual HTML editing. We recently eliminated a couple of steps by integrating Eventbrite’s data API with a platform that generates the HTML emails, taking email prep time down to about 10 minutes and eliminating a lot of potential bugs and omissions.

Ugh. Have fun editing agent information.

The first step was figuring out how the event coordinators currently worked with email. It wasn’t pretty – they were cracking open HTML email templates I’d built previously, modifying the copy and event information, and sending them out. Our email sending platform prepends tags to links, making even simple tasks like editing anchors problematic. At one point, we managed to create an email that previewed fine in a browser, but after parsing by our system it rendered without the entire bottom half. While the vast majority of emails had no issues, what we were doing was complex and hardly ideal. This started a discussion: we already had a great deal of data in Eventbrite, didn’t have a short-term solution for building our own events platform, and really wanted to lock down the HTML to prevent avoidable errors.

It wound up being fairly simple to put together. We’d build a list of our markets, hit Eventbrite’s jQuery API, and pull down all the upcoming events from the selected area. Eventbrite gives you a ton of data, but most of what we wanted was in the event description – and would have been tough to parse and jam into an email. Instead, we scrape the event titles for keywords that let us figure out what type of event has been selected. Once that’s determined, the editable fields (and some uneditable ones, like location, date, and time) are prepopulated with some boilerplate information. At this point the events team can modify the content, toggle some features – do we want to promote an upcoming event? will there be food at the class? – and they’re ready to go.

Adding an instructor in the generator is as simple as searching for their name or email.

But what about solving the busywork of getting agent information? We have hundreds of agents in markets all over the country, and nobody wants to spend 5 or 10 minutes per email searching for their latest photo and verifying that their phone number and email are correct. At this point I got together with Ben & Robert on the Commerce team, who build & maintain our internal Agent Tools platform, and got some pointers. They’d just recently implemented type-ahead search, and it would perfectly solve the agent data problem for our emails. With a lot of help from them, and a little bit of tweaking, the generator was updated to pull down agent information and jam it directly into the template. Once we’re done, the email looks like this!

After smashing some bugs that the events team graciously helped find (it took several days of building emails with the beta generator to really nail things down), we had a pretty robust, semi-automated way to create emails quickly and consistently. The events team can now prep and send promotions much faster than before, and we’ve been able to really standardize our emails on a couple of minor variations. Even better, we’re looking into ways to improve and expand automatic HTML generation beyond emails and into other marketing promotions.

In MKMapView,  a region is an area defined with a lat/long coordinate as the center and a span to specify the distance it covers in lat/long delta. Depending on the dimensions of the MKMapView, the region specified may not fit into the view properly.  So there exists a handy method “regionThatFits” which takes a region as an argument and returns a new region that will include the region you pass in and fits properly in the aspect ratio of the MKMapView. In SDK 5, if the region specified is very tiny, which means the MKMapView’s zoom level can’t support it, the regionThatFits method used to return a minimal region that it can zoom in to. However, with the recent SDK 6 release, the “regionThatFits” method returns a region that the map cannot properly zoom to when you give it a tiny input.

In order to prove this finding, we have come up with a simple prototype an app to perform the following steps:

1. Create a MKMapView in an app
2. Call MKMapView’s regionThatFits method with a tiny region and output the returned region from regionThatFits. In our prototype,
   MKCoordinateRegion region = {{37.787096, -122.391},{1.14441e-05, 9.91821e-05}};
   MKCoordinateRegion zoomedRegion = [mapView regionThatFits:region];
3. Call MKMapView’s setRegion method with the region returned from regionThatFits
   [mapView setRegion:zoomedRegion animated:YES];
4. Inspect the MKMapView’s current region and compare with the region returned from regionThatFits. Notice the current MKMapView’s region is larger than the region returned from regionThatFits!

We have filed a bug report to Apple’s iOS SDK team. Hopefully they can address this issue soon, although we don’t rely heavily on regionThatFits returned region to do any computation at this point.

We hope every mobile developer is having fun with the latest Apple Maps! If anyone has insight about this issue, please leave us a comment.

While this approach is very appealing from a developers point of view of harmonizing the programming languages for building our pages, we were concerned about the size of the javascript required to be downloaded, parsed and ready to go before even showing anything to our users. In fact, we ran into this issue before with our map page. It had ballooned so much that it required over a MB of JS to be downloaded before anything was rendered on the page.

So, with the new design and the inevitable addition of more and more widgets, we took some time and figured out a strategy for handling a page that requires a lot of javascript. Not only that, we wanted to prioritize the “above the fold” content to render as fast as possible.

Dojo Parsing

There are a couple approaches to creating widgets with in dojo. One is to use the parseOnLoad flag of dojo to signal to it that it should parse the DOM once dojo itself and all the dojo.require’s have finished. Another is to hook directly into the dojo.addOnLoad function callback and start creating your own widgets. In both of these cases, in a production environment, we will have to require a main profile rollup of all our javascript code.

Institutionalizing Javascript Phases

However, since we wanted to split up the way we did our rollups, dojo didn’t necessary provide us an easy mechanism when certain JS definitions do not exist yet. At first we looked at trapping the errors of missing widgets in parseOnLoad and just re-invoking dojo.parse.parser() once we got the requires but while this somewhat accomplished what we wanted, it wasn’t quite an elegant solution to our problem. It masked other errors that might have happened with the widget creation.

So we looked deeper at the parser and started pulling out parts it so we could control when and where we wanted to invoke the parser.

A different Dojo type parser

We decided we would divide up our widgets so that the top of the fold widgets would be in one grouping, the middle content in another grouping and any leftover widgets in the last grouping. Internally we call these groupings “phases”. It’s arbitrary in how many phases we have and what widgets we assign to what phase, but we divided them into three main phases so we could minimize the amount of network requests while still having the top of the fold and middle fold content render as early as possible.

The order in which this parser works is pretty simple. First, it will start executing once dojo.addOnLoad invokes our callback. This essentially means for us that it’s pulled down our main dojo.js rollup and the first “phase” rollup. This enables us to start bootstrapping the page by sending out data requests (see our data loader post later!) and then for the leftover javascript rollups. The key here is that we don’t eval the javascript yet, just store it until we’re ready. It may be the case that the javascript is already cached but we still want to render the page in a certain order so that the users will see the top of the fold content first and as early as possible.

So, after we’ve sent out our network requests, we can start the first widget creation phase. The majority of our widgets extend from a widget that is aware that we’re loading our javascript in phases. This widget overrides the buildRendering phase. We disable _Templated’s scanning for children and just inform our parser that the widget it just created has children that need to be created. The parser does this for each widget it creates, collecting each new dojoType within the templates of the widgets. When it collects these types, it first checks to see if it can create it (does the class exist) and if not, saves it off to be reparsed in the subsequent phase loading.

Once the parser finishes up its first phase scan and widget creation, it will invoke the startup() method on all the widgets. This is pretty much inline with the traditional life-cycle definition of startup(). After all the widgets have been informed, the parser will eval the next phase of javascript and loop through all the saved off dom nodes that it did not find the dojo class for yet.

Upon reaching the last phase, if there are still classes left over, we put in some debugging statements that help us identify any dojoTypes that were never created. We also adjusted the creation of the widgets to not blow up the whole page meaning that whenever there is an error while creating a widget, it will be removed from the list and move on.

All this gives us the ability to render each of our phases, attach them to the DOM and present to the user our content as soon as possible.

_delayedAdopt

The last little detail with this setup is programmatic creation. We had to get a little clever so that we could create something once all the javascript rollups have been downloaded. Inspired by Higgin’s article for _Adopt (something that we also use for programmatic creation), we created a second function that acts just like adopt() but uses a string for the class name.

delayedAdopt: function(/* string */cls, /* object? */props, /* dom node */node, /* function */callback) {}

The callback is automatically bound to the context it was created in just like the normal adopt() but will not execute until that the require for that widget has been pulled down. This lets us make the creation of widgets that are not necessary in the top of the fold content be created in a later phase.

Phased loading

With all this together, this gives us the ability to split up our javascript into chunks and force an order of creating the page. It reduces the initial page size down so that the user is interacting and seeing something almost immediately. Most people eat with their eyes and dislike pages that seem to be loading slowly. Hopefully this makes it seem like the page loads faster for users.