Ryan Dahl Introduces Node.JS

Ryan Dahl, the creator of a high-performing web server written in JavaScript, came by Redfin’s San Francisco office to talk about his creation, Node.JS. It was a very funny, thoughtful talk, particularly because Ryan is somehow both opinionated and careful with the truth. He is the latest in a long line of speakers for Engineer-to-Engineer, a series of technical talks hosted by Redfin, Digg, Pandora and Greylock on topics such as Hadoop, Scala, HTML5, Cassandra and Clusto.

Ryan’s presentation is here, and below is a summary of what he said.

This is going to be very introduction-level, with apologies to anyone who has dived deeper.

The goal of Node is to do easy network programming, to be able to create servers and clients that can be thrown together in a fairly simple way, using JavaScript.

Node.JS is a set of C++ bindings for network I/O and socket I/O. The strong focus is on putting together network servers.

Node is a command-line tool. You need to compile it. There are no binaries available. It’s something that runs from your terminal. It doesn’t have any dependencies other than Python to build it.

Let’s understand it by example… The first example is a program that prints Hello and then in 2 seconds, says World.

1. setTimeout(function () {
2. console.log(’world’);
3. }, 2000);
4. console.log(’hello’);

Node has a lot of browser-like APIs. When you’re in JavaScript, you expect it to be Browser-ey, even if it’s not Browser…ish, that is, even if it doesn’t run in the browser.

Node exits automatically. The program drops out when there’s nothing else to do. If there’s a callback pending it keeps running. In the example, after World, the program exits.

Now let’s make this more complicated. What if we want Hello every half second, then on an interrupt signal we want the program to print Bye?

1. setInterval(function () {
2. console.log(’hello’);
3. }, 500);
4.
5. process.on(’SIGINT’, function () {
6. console.log(’bye’);
7. process.exit(0)
8. });

In the browser your central object is the window; in node it’s a process. This global variable exists always.

It’s like a browser listening for a click event. And it’s also like a UNIX program in that you have to end the program. The process object emits an emit when it receives a signal; you only have to listen for it. You can get the pid, the program arguments, you can grab memory usage, you can get the executable path.

A TCP server emits a connection event, whenever someone connects, it says connect, and then it connects.

Now let’s create an event…

1. net = require(“net”);
2.
3. s = net.createServer();
4.
5. net.on(’connection’, function (c) {
6. c.end(’hello!’);
7. });
8.
9. s.listen(8000);

You can load a module; browser-based JavaScript doesn’t support this. You create a server in line 3, in line 5 – 7, we add an event listener, and then finally on line 9 you set up a port so the server is actually listening.

File I/O is non-blocking too. Node does File I/O. Here’s a program that outputs the last time /etc/passwd was modified.

1. var stat = require(’fs’).stat;
2.
3. stat(’/etc/passwd’, function (err, s) {
4. if (err) throw err;
5. console.log(’modified: %s’, s.mtime);
6. });

If you’re on a server being hit by thousands of people, you can’t just wait for the disk to spin, so Node takes the pragmatic view that you should never wait for something to happen. Set up the action to occur, but don’t wait for this action to occur. Give a callback and then drop back. There are two parameters. There’s an error object if the file is not there. Otherwise, you print out the time modified.

Node can do HTTP too. If it was just TCP and file stuff, that would be very limiting. Load the HTTP module; it is called every time you have a request, it writes to the response the header and Hello and World.

1. var http = require(’http’);
2.
3. var server = http.createServer(function (req,res) {
4. res.writeHead(200, {’Content-Type’: ’text/plain’});
5. res.write(’Hello\r\n’);
6. res.end(’World\r\n’);
7. })
8.
9. server.listen(8000);

The HTTP response is chunked because we don’t know how long it will end up being, so we can’t put a Content-Length header at the top. Node is very good at streaming: we’re not limited to “Here’s this movie, buffer it all.” Node streams up to memory, down to disk.

Here’s a streaming HTTP server… it can stream responses without introducing a large amount of weight, you don’t use a thread for each of these. If you curl it, you get Hello, then two seconds later, you get World.

1. var http = require(’http’);
2. server = http.createServer(function (req,res) {
3. res.writeHead(200, {’Content-Type’: ’text/plain’});
4.
5. res.write(“Hello\r\n”);
6.
7. setTimeout(function () {
8. res.end(’World\r\n’);
9. }, 2000);
10.});
11.
12.server.listen(8000);

This is low-level. It allows streaming requests, and requests can be hung while waiting for other things. With AJAX, connections are continually asking “Do you have anything new?, which can be very taxing on the server. Long polling, on the other hand, only involves asking once and then getting a response when the server wants to send you one.

Node’s HTTP server is enabled by the HTTP parser. You can check out http://github.com/ry/http-parser

You might be thinking: “HTTP, Jeez, how hard could it be, it’s a simple protocol.” You’re wrong. HTTP in the real world is extremely complicated. It’s difficult to be able to parse the headers and be able to expose this streaming nature without buffering. This HTTP server buffers nothing. It’s totally callback-based.

The HTTP server only uses 28 bytes per HTTP connection, which is important when you have 1,000 people chatting on a server. 28 bytes is acceptable for overhead; 4 megabytes isn’t.

Now let’s do inter-process communication with other processes. In this example, you pull out the child process. This is something that can spin the disk. Your CPU is much, much faster than the disk. Don’t wait for the disk.

1. exec = require(’child_process’).exec;
2. exec(’ls /’, function (err, output) {
3. if (err) throw err;
4. console.log(output);
5. });

It’s worth nothing that Node never forces output buffer. You can also stream data through the standard in and out of a child process.

Now we spawn the program cat, and we get a reference to that program. Whatever you send to cat, it sends back. You type in Hello, wait 2 seconds, then type Bye. You get Hello, then wait 2 seconds, then get Bye.

1. spawn = require(’child_process’).spawn;
2.
3. cat = spawn(’cat’);
4.
5. cat.stdin.write(’hello\n’);
6.
7. setTimeout(function () {
8. cat.stdin.end(’bye\n’);
9. }, 2000);
10.
11. cat.stdout.on(’data’, function (d) {
12. console.log(d);
13. });

Connecting streams is common. Where I want to go with Node is thinking of everything in terms of streams. There’s standard in and out, there’s file streams, HTTP connections. But mainly we deal a lot with streams. Generally we’re proxying streams and modifying them in the middle.

So this is JavaScript outside the browser. Yes! That’s almost what everybody wants. We’re interacting with the OS in a browser-like way.
We have an HTTP library for streaming. But wait there’s more… here’s a contrived but interesting web-server benchmark. We’ve set up four web servers. They’re all going to respond with a 1 megabyte file. 100 concurrent clients connect.

• Node can handle 822 reqests per second
• Nginx (web server written in C, popular with the Ruby crowd, consider this as good as it gets): 708
• Thin: 85
• Mongrel: 4

This should be shocking to you. You should be urinating right now. Or getting angry. It shocks me.

There are some caveats. NGINX peaked at 4mb of memory, and Node 60mb of memory. I also didn’t sit down for hours and try to make NGINX fast, as I did with Node.

There are a lot of places in Node where the opposite is true, where it sucks while everything else is good. SSL for example.

Node is written on Google’s V8, the JavaScript engine in Chrome. V8 is a masterpiece of engineering. Google took the 14 best VM engineers and locked them in a closet in Denmark. They were given the JavaScript spec and then told to make it fast.

It’s an amazing VM. Much better than Ruby or Python. Incomparable. Or comparable I guess… All these callbacks must seem weird to you but that is where our speed increase comes from.

Result = query (‘select * from T’); //use result

If you’ve done traditional web programming, you’ve probably used activerecord and you access some record. You use a function to do the I/O, but what does your software do while it’s accessing the database. In many cases, nothing. It’s the year 2010, we’re using Rails, and when you access a database, it stops, the world stops for who knows how long, the database might be in LA, and it takes 2 seconds to respond.

To mitigate that, we load balance with multiple processes, all waiting 2 seconds. That’s a form of concurrency to be sure, I guess that’s what processes are for.

When you access stuff in the CPU, it’s very fast. You can assume any operation to take zero amount of time, until you access the disk or the network. It’s not appropriate to treat operations in the CPU in the same way as operations on disk or I/O. Abstracting I/O as a function doesn’t make sense when the time-frames are so different.

• 3 cycles for L1
• 14 cycles for L2
• 250 cycles for RAM
• 41M cycles for disk
• 240M cycles for network

It’s unacceptable to wait for the database when you’re serving many clients. You can fork a thread – it’s hard in Ruby because its threading system is utterly crap, but Java can – so when one thread blocks while accessing the database, you can start new threads. That’s fine. But you can’t use an OS thread for each socket when you want good concurrency. Threads have weight to them, and context-switching is costly too. Each thread takes 4 meg of memory, which is a lot when you have 1,000 concurrent users.

The alternative to using threads is to structure your code like this:

Query (‘select..’ function (result) { //use result });

Node is fast because it never blocks on I/O. And JavaScript is great for this. In Ruby there’s EventMachine, in Python there’s Twisted, somehow it doesn’t jive, you sit down to write the code and it doesn’t work the way that programming language is meant to work, it doesn’t work with all the modules out there – like a MySQL library — to do I/O. But the browser was already set up to be an event loop. Brendan Eich was a genius. Yes it does one thing at a time, but also many things very quickly, because you never block on I/O.

And there’s a culture of JavaScript, an entire generation of programmers who grew up programming browsers, and now they can code on a server, without forking a thread and blocking on except. Java people on the other hand find this callback concept difficult to grasp. “What do you mean? What is it doing while it’s doing nothing?”

Node jails you into this evented-style programming. You can’t do things in a blocking way, you can’t write slow programs.

Node consists of 3 C libraries: V8; event loop (libev) so you don’t have to write something for every OS; a thread pool (libeio), which is necessary for file I/O. There’s a layer for bindings, C++ glue, then the standard library is written in JavaScript. It’s not a thin binding to a C web server, it actually goes through a lot of JavaScript – that’s impressive – V8 is up to the task. I used to write web servers in Ruby, it was awful, every line of Ruby hurts performance; it’s a beautiful language, but a crappy virtual machine. V8 is not that way.

JavaScript can only access the main thread, the C layer has access to blocking functions – we don’t want to have a global interpreter lock – let’s let the experts have access to the threads. To use the threads, program in C.

I wouldn’t use Node.JS to make big websites, but it is one of the only solutions for making real-time, long-polling things. You’ll probably have a bunch of Rails servers and one Node server for a specialized function. As frameworks mature, you can use Node to build the whole website. You won’t have to load-balance it because it’s very fast but you’ll probably have to put it behind a web server, because you don’t trust it, or because SSL support still sucks. The bottleneck will be your gigabit connection into that machine, not memory or anything else.

And that was it! Many thanks to Ryan for a dazzling talk, and to everyone who came. Thanks too to Greylock, Digg and Pandora for helping us put on the event…

Introducing MultiMarker: The fastest way to add many hundreds or thousands of markers on Google Maps

At Google IO 2009, our fearless leader Sasha Aickin (my boss) demonstrated our high-performance Google Maps utility library to the world; we provided directions explaining how to code it, but we didn’t actually ship the code.

Today, I’m proud to announce that we’ve made our MultiMarker utility library code (formerly known as SuperMarker) available to everyone under the Apache License 2.0. As far as we know, it’s the fastest way to add many hundreds or thousands of markers on Google Maps.

HOW FAST IS IT?

Below, we compare the time required to add 1000 markers in four ways:

1. V2 map using the default GMarker
2. V2 map using Pamela Fox’s MarkerLight
3. V3 map using the default google.maps.Marker
4. Using our MultiMarker library

Note that the GMarker timings you see here were recorded by stopwatch, NOT by automated timer. Also note that these browsers were running on different machines, so you shouldn’t use this table to compare browsers with each other; just compare the columns within each row.

As you can see, a speedup of 10-100x is possible using the MultiMarker technique.

Try the examples for yourself:

Google Maps V2 Comparison Test
Google Maps V3 Comparison Test

By the way, you may notice that the data above shows that adding markers to V3 maps is considerably slower than adding markers to V2 maps. This should come as a bit of a surprise to those who watched Stepping up: Porting v2 JavaScript Maps API applications to v3 at Google IO 2010, where Google demonstrated improved performance in V3 vs. V2.

Unfortunately, it appears that the timer they used was incorrect, which you can see for yourself by trying their examples online, especially in IE.

Google’s V2 Speed Test
Google’s V3 Speed Test

In these examples, a number appears on the screen, but the markers haven’t actually appeared yet. It’s especially apparent in the YouTube video, at 11:04.

From the user’s perspective, if the marker isn’t on screen, the map isn’t done drawing yet, so I measured visual performance the only way I know how: using a stop watch. (Fortunately, when you’re counting 11+ seconds, a stop watch is precise enough to measure.) MarkerLight and MultiMarker don’t have this problem, because they draw their icons all at once, so I just trusted the automated timer for those numbers.

If you’ve been negatively impacted by slower markers in Google Maps V3, MultiMarker may be a good fit for your needs. Give it a try!

Service Oriented Architecture with Varnish and Edge Side Includes

As we talked about before, Redfin uses Varnish to implement Edge Side Includes (ESI.) This involved breaking a single big (and expensive) page into individual chunks; each chunk would be generated by separate code, and would be cached on a different schedule.

Once we broke our expensive page into chunks that could be individually cached, it seemed pretty easy to have those chunks served up by different backend servers. Voilà, a monolithic app became “service oriented“! This would let us run the different software components on different machines (with different performance characteristics, different SLAs, even implementations in different languages/environments!)

Of course, nothing is actually that easy, and we made a number of mis-steps before we figured out how to do it.

How To

Varnish allows you to define multiple backends in your VCL. And in your vcl_recv function, you can decide which backend should handle a particular request. At Redfin, we added a new Varnish backend for each of our ESI endpoints, and we added logic to choose the relevant backend by URI. In practice, we actually only have one pool of machines handling our ESI requests, so all of our Varnish backends actually point to the same place.

So the first piece of the puzzle is on our main web servers. On the main web servers, requests go through Varnish. Requests for “normal” pages are sent through to Tomcat, but requests for ESIs are sent to one of the SOA backends. Here’s an example of what the VCL file might look like:


backend default {
  .host = "localhost";
  .port = "8080";
}
backend similars {
  .host = "similars.redfin.com";
  .port = "6081";
}
backend relevantlinks {
  .host = "relevantlinks.redfin.com";
  .port = "6081";
}

sub vcl_recv {
  if (req.url ~ "^/esi-listing-similars" || req.url ~ "^/esi-property-similars") {
    set req.backend = similars;
  }
  else if (req.url ~ "^/esi-listing-trackbacks") {
    set req.backend = relevantlinks;
  }


You might have noticed that the “localhost” backend is associated with port 8080 (where Tomcat is running), but the ESI backends are associated with port 6081 (where Varnish is running on those remote machines.)

We want the instance of Varnish on the main web server to cache content from the main web server, and the instances of Varnish on the ESI backends to cache the content from those backends. This has a few benefits:

• Our effective cache is bigger, since we have caches on multiple machines, each of which has fixed memory
• Having independent caches prevents one set of items from pushing another set out of the cache. If all the data were in a single cache, then cache entries holding similars information (which is small, but expensive to recreate) could be pushed out of the cache by cache entries of “main page” content (which is big and relatively cheap to recreate, but we’d still like to cache.)
• It’s easy to flush individual caches without having to worry about performance problems with other parts of the site

We have another design goal: we’d like to have a single distribution of our software. We’d like to have a single WAR that we can put on any machine; we do NOT want to have to deal with multiple builds, with figuring out which build has been installed on which machine, etc. We’d like to be able to switch a single machine from being a standard web server to being an ESI endpoint without having to redeploy or reconfigure.

This creates a conundrum. We want our main web servers and our ESI servers to be identical, but we also want them to act different. In particular, when an instance of Varnish on a web server gets a request for an ESI fragment, it should redirect that request to an ESI server (more precisely: to the Varnish instance running on an ESI server.) But when an instance of Varnish on an ESI server gets a request for an ESI fragment, it should forward the request to the local Tomcat instance. It should NOT forward the request to ITSELF. Forwarding port 6081 to port 6081 creates an infinite loop- not good.

We want to break the symmetry between the standard web servers and the ESI servers, and we do that by messing with the URIs.

We prepend our ESI URIs with a known prefix, which means “forward this to the ESI server.” But when we process the URI (while forwarding it), we strip off that prefix, so that the ESI server does not also forward it to itself. That’s harder to say than it is to code. The VCL code looks like this:


sub vcl_recv {
  if (req.url ~ "^/backend/") {
    set req.url = regsub(req.url, "^/backend/", "/");

    if (req.url ~ "^/esi-listing-similars" || req.url ~ "^/esi-property-similars") {
      set req.backend = similars;
    }
    else if (req.url ~ "^/esi-listing-trackbacks") {
      set req.backend = relevantlinks;
    }


This breaks the circularity. The path of requests looks like:

1. A requests comes into Varnish on the standard web server for /path/to/a/page
2. Varnish forwards the request to the local Tomcat instance
3. Tomcat responds with HTML that includes <esi:include src=”/backend/esi-listing-similars” />
4. Varnish processes the ESI, and must make a request for /backend/esi-listing-similars
5. The Varnish instance on the standard web server strips off “/backend”, and sends a request for “/esi-listing-similars” to the ESI server
6. The Varnish instance on the ESI server gets the request for “/esi-listing-similars”
7. Since there’s no “/backend” prefix, the Varnish instance on the ESI server forwards the request to its local Tomcat instance
8. The Tomcat instance on the ESI server processes the request, and responds with the relevant HTML fragment
9. The Varnish instance on the ESI server caches the HTML fragment and returns it
10. The Varnish instance on the standard web server parses the HTML fragment into the main page content and returns it to the browser

This example points out another tricky bit- how do we assure that the HTML fragment is cached by the Varnish service on the ESI server, but not by the Varnish service on the standard web server? To handle this correctly, we add a header to the response which indicates if it’s already been cached:


sub vcl_fetch {
  if (req.url ~ "^/esi-") {
    if (obj.http.X-RF-Cached ~ "true") {
      pass;
    }
    set obj.http.X-RF-Cached = "true";


This code says “If there’s an X-RF-Cached header present, then don’t attempt to cache. If there is NOT an X-RF-Cached header present, then add one, and attempt to cache.” With this addition, the HTML fragments will only be cached on the first Varnish instance they pass through, which is on the ESI server in our case.

How NOT To

The solution described above works, and meets our requirements. But we also tried some solutions that did NOT work. Perhaps you can learn from our failures…

Putting Absolute URIs into ESI Includes

Our first thought was that we’d put absolute URIs into our ESI includes in the HTML. For instance, we tried to put <esi:include src=”http://similars.redfin.com:6081/esi-listing-similars” /> into the main HTML of our page. Varnish simply (and correctly, I think) ignores the host name and port. Including http://similars.redfin.com:6081/esi-listing-similars will cause Varnish to act as if you included /esi-listing-similars, and Varnish will use whichever backend it thinks is relevant, regardless of the host name or port in the URI.

Using a Single Server as both a Standard Web Server and an ESI Server

When doing testing, or when some of our servers were unavailable, we were tempted to use a single server as both the standard web server and the ESI server. It seemed like this should work- the trick with the “/backend” prefix should prevent infinite circularity. However, it didn’t work. It seems that Varnish is doing its own checks for circularity, and noticing that a single request passed through the same Varnish instance multiple times (which NORMALLY would be a problematic example of circularity, but we’ve got our clever symmetry breaker in there!) Anyway, Varnish doesn’t allow it, and causes those semi-circular requests to fail.

P.S.

Thanks to D’Arcy Norman for the photo!

Engineer-to-Engineer: Evolving a New Analytical Platform with Hadoop

In the second installment of our San Francisco series of engineer-to-engineer lectures, Jeff Hammerbacher described the challenges of building data-intensive, distributed applications and how using Hadoop saved the day at Facebook.  Speaking to an audience of approximately thirty Hadoop experts and enthusiasts hailing from all around the Bay Area, the Valley, and even Seattle, he also discussed what’s wrong with today’s analytical platforms and what will shape the platform of the future.

And Jeff should know.  After studying Mathematics at Harvard and wearing a suit as a quantitative analyst on Wall Street, he conceived, built, and led the data team at Facebook.  He then went on to start Cloudera, the leader in commercializing Apache Hadoop, where he currently works as Chief Scientist and VP of Products.  Jeff also served as Contributing Editor for a book: Beautiful Data: The Stories Behind Elegant Data Solutions, the proceeds of which are split between Creative Commons and Sunlight Labs.

The Scoop on Hadoop

Hadoop is an open source framework that enables data-intensive distributed applications to efficiently process gigantic amounts of data.  It’s an open source implementation of the MapReduce approach to processing data.  MapReduce was invented at Google to deal with the massive quantities of data necessary to index the web.  There are two main components to the system: the Hadoop Distributed File System (HDFS) which stores and maintains data across many machines, and the MapReduce engine which processes the data.

But the talk didn’t really go into Hadoop internals — as Jeff pointed out, the documentation is readily available online.  Rather, the talk was about how and why Hadoop will provide the foundation on which the next generation platform for analytics will be built.  Making bold predictions about technology is hard.  Jeff quoted Larry Ellison’s quip that “the computer industry is the only industry that is more fashion-driven than women’s fashion.”  And yet, using real-world examples from his experience at Facebook, Jeff makes a compelling sell.

Bottlenecks, Costs, the Black Box, and the Kitchen Sink

A typical architecture for large-scale data analysis includes a data source, a data warehouse, ETL (aka: “extract-transform-load”; the step that gets data out of and into RDBMSs and converts source data to the data warehouse’s format), and business intelligence and analytics systems – all of which are usually centered around relational databases.  However, Jeff stressed that a relational database is a specialty and not a foundation, arguing that the abstractions provided by them are no longer useful on their own for analytical data management.

One reason is that over the past few years, there has been an explosion in data volume primarily originating from machine-generated logs.  By simply tweaking an Apache log, you can grow your data volume and complexity by several orders of magnitude.  As we’ll see in Facebook’s case, their relational database approach simply didn’t scale and they soon needed new tools to handle the load.

Another point Jeff made was that the percentage of data that actually gets stored in a relational database is shrinking.  What do you do with all the unstructured data (accounting for 95% of the digital universe) that doesn’t necessarily make sense to persist relationally?  Do you still need expensive relational data warehouses and proprietary boutique servers?  Jeff’s team at Facebook made a bet on commodity hardware which turned out to be a good move, ultimately pushing the complexity out of hardware and into the software layer.

They also bet on open source data stores built by consumer web firms, arguing that web properties have the most representative problems: scalability and unstructured data management.  Jeff stated that most production-quality data stores came from enterprise software firms in the mid-1990s, but now a growing percentage of the world’s data is persisted in open source data stores.  He also mentioned that a nice side-effect of adopting open source solutions is that it’s much better to have a modular collection of open tools rather than an opaque abstraction.  Why?  Because there’s great benefit in being able to pick and choose solutions and understand what’s going on under the hood.

Jeff noted that another problem is that, in many cases, enterprise software does not service developers well. Many relational data warehouses simply just expose SQL; but to get real traction/adoption from developers, you need more than that…  You need open applications for analysis, not just a SQL interface.  He feels that “in addition, these data stores often expose a proprietary interface for application programming (e.g. PL/SQL or TSQL), but not the full power of procedural programming.  More programmer-friendly parallel dataflow languages await discovery, I think.  MapReduce is one (small) step in that direction.”

Where is this new platform going to come from?  Any new platform must be centered around addressing these new user needs, which is hard to achieve by re-implementing an old spec in a new, clever way.  He cautioned that implementing a new, successful cut of the ANSI SQL spec would be a real undertaking.  Not only would it take ages before you had anything to show, but it would likely suffer the same scalability problems of previous implementations.

Facebook and Hadoop are Now Friends

Using Facebook as a real-world example, Jeff described the challenge of measuring how changes to the site improved or impaired user experience.  Their original data analysis system featured source data living on a horizontally partitioned MySQL tier and a cron job running Python scripts that pinged stats back to a central MySQL database.  The main problem with this setup was that it made intensive historical analysis difficult since the source data was spread over many machines and aggregating the data to the analytics database was a slow, inefficient process.  Plus, when it barfed, it took three days to replay the edit logs in order to diagnose the problem.

So Facebook hired a data warehouse engineer to build a 10TB Oracle warehouse.  This worked for a bit and would’ve been fine for small and medium-sized businesses, but ultimately didn’t scale — particularly when they turned on impression logging which generated over 400GB of data on the first day!  This quickly grew to 1TB of data per day in 2007.

You might suggest that since disks are cheap, why not throw more storage at the warehouse?  It turns out that, in addition to the problem of data volume, there was also a bottlenecking CPU utilization problem. The ETL process ended up taking more than a day to aggregate, import, and load the necessary data for analysis.  Jeff went on to explain that proprietary ETL vendors have lots of downsides and generally don’t scale well for large sets of databases (on the order of thousands, in Facebook’s case).  In addition, when “warts” start to show up in proprietary vendors, the closed nature of the software prohibits developers from tinkering with the source to diagnose and resolve problems.

Meanwhile, his team started to play with Hadoop on the side as an open source alternative.  They got a Hadoop cluster to replace the data collection and processing tiers.  So the new architecture still has multiple data sources (log files, MySQL) but is now fed into HDFS instead.  Work is done via MapReduce and the artifacts are then published to Oracle RAC servers for consumption by business intelligence and analysis.  It also simultaneously publishes results back to the MySQL tier.

From “Facebook’s Petabyte Scale Data Warehouse using Hive and Hadoop“, slide 21

Initially, this shift was met with a lot of resistance mostly because Hadoop is Java-based and, since the majority of Facebook’s services were written in C++, the developers there weren’t comfortable in Java. But it wasn’t long before the new platform showed its strengths:

• Switching to this system greatly reduced latency because the ETL process is no longer done in flight – it’s done after persistence in Hadoop.
• Hadoop enabled Facebook to efficiently crunch extremely large data sets on the order of multi-petabytes, previously impractical under the old system.
• The Hadoop data warehouse became easily accessible to developers which turned out to be a real bonus.  Developers previously found SQL to be an unfriendly environment because they couldn’t predict the impact of running SQL (it was easy for them to hose themselves and others) and because the dev environment for SQL was crude.  After switching, however, they found that a lot of Facebook’s developers started freely playing with the data set which fostered innovation and led to new features.

Shaping a New Platform

Jeff emphasized that while Hadoop provides a great foundation for data analysis, it’s not the whole story.  Today, there are many technologies built on top of Hadoop that need to be considered for your system.  For example: there is Hive, a system for offline analysis; there is HBase, an open source implementation of Google’s BigTable to name a couple.  He remarked that the abstraction layer needs to be redrawn to include the functionality provided by ETL, master data management (MDM), stream management, reporting, online analytical processing (OLAP), and search tools; all with a unified UI.

Jeff explained that SQL Server 2008 R2 is a good model.  SQL Server is no longer just a database – there are a bunch of associated products in the box offering a full suite of features.  You still have the old features like SQL Server Integration Services (ETL), SQL Server (data warehouse), SQL Server Reporting Services, SQL Server Analysis Services, and full-text search.  But now you also get a bunch of new features such as stream management (StreamInsight) providing real-time analytics, OLAP (PowerPivot) enabling rapid navigation of subsets of data, collaboration via SharePoint, MDM for integrating disparate data sources and entity resolution, and features that aid in scaling your servers out to a many-node SQL solution.  Jeff remarked that it’s “kind of scary that Microsoft has started to do a lot right within the last 5 years.”

Providing a full suite of features is also what Cloudera does well, but for Hadoop.  They’re not the primary developers of this stuff (currently only 3 out of 17 contributors on HDFS), but they do an excellent job at packaging and polishing Hadoop and make their money in training, services, and support.  And, like Microsoft, they eat their own dogfood: using the tools they build to solve their own business problems.  Jeff joked that it’s “interesting being a vendor now – I can see what we put these other vendors through [while at Facebook].”

Many thanks to Jeff for the great talk, Greylock for helping with the logistics (and providing the delicious pizza and beer), and to everybody that came out!  Be sure to check out the next talk on June 10^th when our own Sasha Aickin, Redfin’s head of user experience, will weigh HTML 5 vs. Native Apps.

Synchronous/Asynchronous Switching with Varnish

When your webapp is serving up content that’s expensive to generate, you may want to serve it up asynchronously- via AJAX calls. This is particularly appealing when content is “below the fold.”

However when that content is cached, you want to serve it up as quickly as possible. If you’ve already calculated the content, you’d like to include it inline in the page, without requiring an AJAX roundtrip. That way, you avoid the latency of an unnecessary round-trip. You also allow the page to be fully rendered (so content doesn’t jump around), etc.

You can optimize for the empty cache, or you can optimize for the full cache, but it seems hard to optimize both experiences.

Redfin faces exactly this conundrum with our listing pages (e.g. http://www.redfin.com/CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622.) Calculating the Similar Listings and Similar Sales is expensive and performed in real time. We cut this Gordian Knot through the use of the Varnish caching reverse proxy, along with clever use of ESI (Edge Side Includes.) For an overview of how we use Varnish at Redfin, see our previous post.

We want to say “if there’s a cache miss, then do AJAX, but if there’s a cache hit, then just include the content.” We have to make sure that the AJAX calls will fill the cache, such that subsequent requests will see cache hits, of course!

I’ll outline what the requests/responses look like for us, then I’ll include some pseudocode that supports this.

At the beginning of time, the cache is empty, and the browser requests information on a Listing.

That’s all great, but we still haven’t used the cache! The cache entry will get used for subsequent requests for the same page, like this:

There are two things worth noting about this exchange.

First, when the backend server gets a request for /similars?property_id=604622, it has to decide if it should be returning the real HTML, or should be returning Javascript that will retrieve the HTML via AJAX. It makes this decision based on the value of a header passed in by the client. When the client is making an AJAX request, it knows it better NOT get back a response that generates AJAX requests (that’d be a death spiral.) Therefore, when it makes the AJAX request, it includes the special header. In all other cases, the special header is NOT included. When the header is included in a request, the server will generate the real HTML. When the header is not included, Varnish may answer the request from cache, or it may pass through to the backend server. If the request is fulfilled by the Varnish cache, then it’s the real HTML, but if it’s fulfilled by the backend server, it’ll be the AJAXy HTML.

Second, there are two URLs that have to do with similars.

/similars?property_id=604622 is an internal-use-only URL that returns the content (either the proper HTML or the AJAX code.)

/extranet-similars?property_id=604622 is an externally facing URL that only returns an ESI fragment (which will subsequently be filled in by Varnish. This way, the ESI endpoints are never available to the extranet; Varnish can get to them, but extranet clients have no need for them. This lets us be lazy with the ESI URLs. For example, URLs that are exposed to the extranet do extra validation to check if the user is logged in, etc. URLs for internal use only, such as the ESI URLs, can skip that work. This also lets us change the URLs when the property changes, to facilitate cache busting (see the “Cache busting” section in ESI and Caching Trickery in Varnish for more information.

Pseudocode

OK, so we know what we want the interaction to look like. What code will make this happen? Here’s some Javaish pseudocode that illustrates how it might work:


/*
Invoked for requests like http://www.redfin.com/[address]/home/[property id]
*/
public void handlePropertyRequest(Request request, Response response, long propId) {
   Property property = getProperty(propId);
   response.write("<html><head></head><body>" +
      ...
      "<esi:include src='/extranet-similars?property_id=" +
         propId +
         "&last_mod=" +
         property.getLastModified() +
      "'/>" +
      ...
      "</body></html>");
}



/*
Invoked for (extranet) requests like /extranet-similars?property_id=[property id]&last_mod=[date]
*/
public void handleExtranetSimilarsRequest(Request request, Response response, long propId) {
   Property property = getProperty(propertyId);
   response.write("<esi:include src='/extranet-similars?property_id=" +
         propId +
         "&last_mod=" +
         property.getLastModified() +
      "'/>");
}



/*
Invoked for (intranet) requests like /similars?property_id=[property id]&last_mod=[date]
*/
public void handleSimilarsRequest(Request request, Response response, long propId) {
   if (null == request.getHeader("full_html")) {
      //This request does NOT demand that we return the actual HTML.
      // We will return a script block that will fetch the HTML via AJAX.
      response.write("<script>" +
         "dojo.addOnLoad(" +
            "function() {" +
               "dojo.xhrGet({" +
                  "url: 'http://www.redfin.com/extranet-similars?property_id=" + propId + "'," +
                  "load: function(response, ioArgs){" +
                     "dojo.byId('similar_homes').innerHTML = response;" +
                     "return response;" +
                  "}," +
                  "headers: {'full_html': 'true'}," +
                  "handleAs: 'text'" +
               "});" +
            "}" +
         ");" +
         "</script>");
      //Do NOT cache the script
      response.setCacheable(false);
   }
   else {
      //This request wants the actual HTML for similars
      response.write(getSimilarsHTML(propId));
      //The similars HTML is cacheable- that's the whole point!
      response.setCacheable(true);
   }
}


ESI and Caching Trickery in Varnish

Varnish is a high performance, flexible, open source HTTP accelerator.

We started using Varnish at Redfin in our last major release, a few weeks ago. It’s pretty much invisible to our end users, but we’re so happy with it that we wanted to give the folks who made Varnish their props in public. It has really been great!

Varnish combines three technologies that are really useful at Redfin:

1. A caching reverse proxy to reduce load on our backend servers
2. ESI (Edge Side Includes) to break a page into snippets of HTML which can each have their own caching strategy
3. VCL (Varnish Configuration Language) which enables fine grained control of Varnish

We use Varnish to accelerate the delivery of home details pages. When you visit the page for a home (e.g. http://www.redfin.com/CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622), parts of that page are cacheable but other parts can’t be easily cached. For example, the description of the home may be available to all users, but MLSs require us to hide some historical information from users who aren’t logged in. Further, while most of the page might be highly cacheable, the “Sites Linking to 830 El Camino Del Mar” section isn’t as easy to cache- a blog post that refers to our page (via a trackback) may come in at any time.

ESI nesting makes it easy to accomodate these vagaries.

Conceptually, here’s what the HTML for our main page looks like:

<html>
<body>
Some notes about this home

Median House Values:
<esi:include src="/esi-listing-regions?listing-id=123" />
</body>
</html>


Varnish will fill in the details of each of the esi:include sections with results from the “src” URL. In this example, a single HTTP request from the browser to Varnish will cause Varnish to make three HTTP requests to the backend server (one for the main page, one for the trackbacks, and one for the similars.)

Turning a single request into three requests doesn’t really help per-se, but it does enable caching. Previous to ESI, we were unable to cache the page as a whole since the “Sites Linking to” section was uncacheable. By breaking the page into three sections, we can support caching for some of the sections, while disallowing caching of the other sections.

The workflow of a request that’s partially answered from cache might look something like this:

1. The browser requests http://www.redfin.com/CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622
2. Varnish receives that request, and looks up the URL in its cache
3. Varnish finds a match in the cache, so it doesn’t send the request for /CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622 through to the backend. Instead it retrieves the content from the cache, and searches it for ESI tags.
4. Varnish finds the ESI include for /esi-listing-trackbacks?listing-id=123
5. Varnish looks up /esi-listing-trackbacks?listing-id=123 in the cache. There’s no entry, so Varnish requests /esi-listing-trackbacks?listing-id=123 from the backend.
6. The backend calculates the content for /esi-listing-trackbacks?listing-id=123 and returns it (along with cache control headers specifying that the results should not be cached)
7. Varnish likewise retrieves the results for /esi-listing-regions?listing-id=123
8. Varnish knits the three HTML snippets together and returns the results to the browser

The big win here is that ESI allows us to cache the main body of the page, even though the trackbacks cannot be cached. This is a tricky bit, so I’ll repeat it. The “outer” HTML, which is the main body of the page, is cached. But the “inner” HTML, the HTML for trackbacks, is NOT cached. The cache of the outer content doesn’t include the inner content- it just includes a token saying “fill in this inner content before you use this cache entry.”

Of course, that’s just the simplest case. In practice, we faced a number of minor challenges while implementing this.

1. Recording every hit

We have two conflicting goals. On the one hand, we’d like to serve content up from cache as often as reasonable- users get the content faster, and our backend systems scale better. On the other hand, we’d like to record every page hit. Whenever a user views a page describing a listing, we record various information. We would like every request to get through Varnish and into our backend, so that we can record this information.
As with nearly every problem in Computer Science, this is solved by adding a layer of code. In this case, the “outer” request is NEVER cached, but all it does is record the hit and generate an ESI include. The “inner” request does the heavy lifting, but responses are cached. For example, the user might request http://www.redfin.com/CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622 which would result in this “outer” response:

Cache-Control: max-age=0

<esi:include src="/esi-display-listing?cache-for-logged-out&listing-id=604622" />

which would in turn generate a cache lookup for /esi-display-listing?cache-for-logged-out&listing-id=123. If that’s cached, it’s fast. If it’s not cached, we gotta do all the work.

2. Caching public content without caching user-specific content

The main page content for a home (e.g. http://www.redfin.com/CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622) is the same for all anonymous users. However, users that are logged in will see additional details, such as whether or not that home is a “favorite.” Thus, it’s easy to cache for anonymous users, but harder to cache for logged in users (we don’t cache the main page content for logged in users.) It’s easy enough to set the cache-control response headers such that Varnish won’t cache content for logged in users. But we wanted to optimize a bit more- we wanted to avoid even attempting cache lookups when the user is logged in. We did this by adding VCL which examines the incoming request. If the request includes cookies that indicate the user is logged in, we skip the cache lookup. We also put a special token into the URL to make it easy for the VCL logic to know that it should do this magic for the request (since the URLs are ESI URLs, they’re not visible to the extranet.) Here’s what the VCL looks like:

sub vcl_recv {
    ...
    if (req.http.Cookie ~ "RF_AUTH") {
        set req.http._rf_login = regsub( req.http.Cookie, "^.*?RF_PARTY_ID=([^;]*?);*.*$", "\1" );
    }

    # cookies by default make requests in Varnish uncacheable
    unset req.http.Cookie;
    ...
    if (req.url ~ "cache-for-logged-out") {
        #Directive says to use cache for logged out users, but not for logged in users
        if (req.http._rf_login) {
            #Since there's an RF_AUTH, the user is logged in- do not use cache
            pass;
        }
        else {
            #The user is NOT logged in- use cache (but do not look up based on cookies)
            lookup;
        }
    }
    ...
}


3. Cache busting

We’d like to cache HTML describing a listing for a long time (24 hours), but when we get new listing data, we want to show that to users immediately.

One approach is to explicitly invalidate any cache entries that refer to the listing. We could identify all Varnish instances that might cache the data and individually invalidate the content in each one. However, that’s a little difficult to do from Java, it may be unreliable (it requires that we keep good records about all Varnish instances), and it’s generally a PITA.

Instead, we include the last modified time of the listing in the URL. Again, the ESI URLs are internal, so this doesn’t dirty our extranet URLs. My earlier example was incomplete. A request for http://www.redfin.com/CA/San-Francisco/830-El-Camino-Del-Mar-94121/home/604622 might generate a response that looks like this:

<esi:include src="/esi-display-listing?cache-for-logged-out&listing-id=604622&last-mod=1272651333452" />

(note the “last-mod” argument, which represents that last modification date of the Listing.) That way, whenever the listing changes, the URL to the main ESI fragment will change- stale cache entries will be orphaned.

4. Tuning Varnish

When we initially deployed Varnish, we were seeing 503 errors- Varnish was returning 503 Service Unavailable errors. Michael Young (our intrepid CTO) changed many of the Varnish settings, including connect_timeout, sess_workspace, thread_pool_min, and thread_pool_max. The most important thing he did was match the Varnish threads to our expected traffic, and the 503 errors went away (pretty much.)

P.S. Thanks to Odalaigh for the gorgeous image

Engineer-to-Engineer Talk: How and Why Twitter Uses Scala

To kick off our San Francisco series of engineer-to-engineer lectures on new technologies and interesting problems in consumer software, we invited in the Great Alex Payne to talk about how Twitter uses Scala, a programming language that combines traits of object-oriented languages and functional languages with an eye toward supporting concurrency better in large-scale software.

Alex started at Twitter in 2007, working remotely in Washington DC, when there were “only one and a half engineers.” Now, Twitter has 170 engineers. “It has been an interesting process,” Alex said. Right after his talk, Alex packed up his cats and headed for Portland, where he’ll still work for Twitter, but ensconced in a smaller, more closely-knit community. Here are his thoughts on Scala (Alex talks fast, and doesn’t waste many word, so my hands were in a rictus of agony from trying to type what he wrote) :

Best, Glenn at Redfin

I started working the programming interface when we were at this very early stage. Now, it handles a couple billion operations every day. It is being baked into more and more of the Web.

I’ve spent the past year working on Twitter’s infrastructure. For that, we use a weird language called Scala. I worked on a book for O’Reilly about Scala that you could sit down with over a three-day weekend to get up to speed on the language.

Why Use Scala?
Why use Scala when you have Ruby and Ruby on Rails? Well, we still use Rails. It works great for front-end stuff. The productivity is worth the tradeoff for working in a slower-performing dynamic language. When you think about what a web framework is doing under the hood, it’s tons and tons of string concatenation. Ruby on Rails can handle that.

What we had a need for as Twitter grew was for long-running heavy processes, message-queuing, caching layers for doing 20,000 operations a second. Ruby garbage-collection is tough, Ruby doesn’t do really well with long-running processes.

Languages Twitter Considered
We knew we needed another language. How did we pick a language that was really fun for us? We considered Java, C/C++ of course. And we looked at Haskell and OCaml for functional programming, though neither has gotten much commercial use. Erlang developers are doing stuff with a lot of network I/O but not with a lot of disk I/O; the knowledge-base around the language wasn’t great though, and the community seemed inaccessible.

Java is easy to use, but it’s not very fun, especially if you’ve been using Ruby for a while. Java’s productive, but it’s just not sexy anymore. C++ was barely considered as an option. Some guys said, if I have to work in C++ again, I’m going to stab my eyes out with a shrimp fork. Java-script on the server-side via Rhino had performance problems, and it wasn’t quite there yet when we were evaluating it.

So what were our criteria for choosing Scala? Well first we asked, was it fast, and fun, and good for long-running process? Does it have advanced features? Can you be productive quickly? Developers of the language itself had to be accessible to us as we’d been burned by Ruby in that respect. Ruby’s developers had been clear about focusing it on fun, even sometimes at the expense of performance. They understood our concerns about enterprise-class support and sometimes had other priorities.

We wanted to be able to talk to the guys building the language, not to steer the language, but at least to have a conversation with them.

Was Scala Fast?
And did Scala turn out to be fast? Well, what’s your definition of fast? About as fast as Java. It doesn’t have to be as fast as C or Assembly. Python is not significantly faster than Ruby. We wanted to do more with fewer machines, taking better advantage of concurrency; we wanted it to be compiled so it’s not burning CPU doing the wrong stuff.

What Alex Likes About Scala
Scala is a lot of fun to work in; yes, you can write staid, Java-like code when you start. Later, you can write Scala code that almost looks like Haskell. It can be very idiomatic, very functional — there’s a lot of flexibility there.

And it’s fast. The principal language developer at Scala worked on the JVM at Sun. When Java started, it was clearly a great language, but the VM was slow. The JVM has been brought to the modern age and we don’t think twice about using it.

Scala can borrow libraries from Java libraries; you’re compiling down to Java byte code, and it’s all calling back and forth in a way that is really efficient. We haven’t run into any library dependencies that cause problems. We can hire people with Java and they can do pretty well.

The community is small but growing, and it’s really accessible. We got to sit down with Martin and ask him and his team about funding for Scala, how problems with Scala will get solved. We’ve never really had to call on that level of access, but it’s really nice to know it’s there.

The Grand Unified Theory of Scala
The grand unified theory of Scala is that it combines objective-oriented programming (OOP) and functional programming (FP). Scala’s goal is to essentially say OOP and FP don’t have to be these separate worlds. It’s kind of zen, and you don’t get it when you first start. It’s really, really powerful; it’s nice to have a language with a thesis, rather than trying to appeal to every programmer out there. Scala is trying to solve a specific intellectual problem.

You have methods that take anything between a string and several point away on the inheritance chain from a string. The syntax is more flexible than Java; it’s very human-readable, as you can leave out period between method calls so it looks like a series of words. Your program can make nice declarative statements about the logic of what you’re trying to do.

Traits, Pattern-Matching, Mutability
With Scala, you can also use traits. This is handy because of course you have cross-cutting concerns in your application. For example, every object needs to be able to log stuff, but you don’t want everything extending from a logger class — that’s crazy. With Scala, you can use a trait to shove that right in, and you can add as many traits as you like to a given class or object.

You can choose between mutability and immutability. This can be dangerous. 9 out of 10 times you use immutable variables when you want predictability, especially when you have stuff running concurrently. But Scala trusts the programmer for mutability when he or she needs it.

Scala has the concept of lazy values – you can say lazy val x = a really complicated function. That isn’t going to be calculated until the last second, when you need that value. This is nice.

Pattern-matching is nice too. It lets you dive into a data structure so you can, for example, explode out a collection that matches an array with “2” as its third element. You can break out strings and regular expressions, and you can pattern-match groups with regular expressions.

An oddball feature that is really useful is the ability to use XML literals, so that you can make something equal to an XML literal, as if the XML literal is a string. You don’t have to import Sax or some crazy XML library.

The Concurrency Story
When people read about Scala, it’s almost always in the context of concurrency. Concurrency can be solved by a good programmer in many languages, but it’s a tough problem to solve. Scala has an Actor library that is commonly used to solve concurrency problems, and it makes that problem a lot easier to solve.

An Actor is an object that has a mailbox; it queues messages and deals with them in a loop, and it can leave a message on the floor when it doesn’t know what to do with it.

You can model concurrency as messages – a unit of work — sent to actors, which is really nice. It’s like using a queuing system. You can also use Java.util.concurrency stuff too, Netty and Apache Mina, dropping it right in. You can rewrite the Actor implementation, and some folks have gone so far as rolling their own software transactional memory libraries.

Java interoperability is a big, big win. There are ten years of great libraries, things like Jodatime. We use a lot of Hadoop and it has been easy to wire Scala to the Hadoop libraries. We use Thrift, without having to patch it; we use libraries from the Apache Commons and from Google.

How Twitter Uses Scala
So that’s why we use Scala, but how do we use it?

In the enterprise world, a service-oriented architecture is not new, but in Web 2.0 it is crazy new science. With PHP or Ruby on Rails, when you need more functionality, you just include more plugins and libraries, shoving them all in to the server. The result is a giant ball of mud.

So anything that has to do heavy lifting in our stack is going to be an independent service. We can load-test it independently, it’s a nice way to decompose our architecture.

What services at Twitter are Scala-powered? We have a queuing system called Kestrel. It uses a souped-up version of the mem-cache protocol. We originally wrote it in Ruby — it got us through a few weeks, but because Ruby is a dynamic language, the service began to show its performance weak spots.

Flock to Store the Social Graph
We use Flock to store our social graph, as a denormalized list of user ids. It’s not a graph database, so you can’t perform random walks along the graph. But it’s great for quickly storing denormalized sets of user ids, and doing intersections. We’re doing 20,000 operations a second right now, backed by a MySQL schema designed to keep as much as possible in memory. It has been very efficient — not many servers are needed.

Hawkwind for People Search
Our people-search is powered by a Scala-built service we called Hawkwind. It’s a bunch of user objects dumped out by Hadoop, where the request is fanned out to multiple machine and then pulled back together.

Hosebird for Streaming
We stream out tweets to public search engines, using a low-latency, HTTP-based, persistent connection system called Hosebird. We looked at queuing systems that financial-services companies use, but couldn’t find anything that could handle the volume of the load. We built something on top of Jetty using Scala. We have more Scala-powered services in the works that I can’t talk about.

Thrift for Transferring Data
We use also Thrift, built at Facebook then open-sourced at Apache. With Thrift, you can define data structures and methods, and it deals with everything you don’t want to deal with to efficiently represent data and get it from point A to point B. As your system evolves, your method signatures change, and Thrift has a nice system for creating positional arguments and being backwards compatible.

These services make our life a lot easier. We often staff projects with two people who are pair programming, sitting together for six or eight weeks. These guys can build something like people-search in a couple of months.

The only problem with so many different teams is that there is some divergence in terms of operational approaches – we have to work with ops guys to monitor the right stuff, be it disk or memory or what have you — but we can resolve that jitter over time. We’re ok with the tradeoffs.

The Development Environment
OK, now let’s talk about the tools… the IDEs for Scala are not up to snuff, that is true. IntelliJ IDEA is good but it’s shockingly buggy. The solution we’ve settled on is just using a plain text editor. We use EMACS, as there’s a really nice mode for the build tool. That takes compile/test BS out of your workflow. Of course, you can give the IDEs a try. Even though I’m an IDE cynic, maybe they’ve improved; that said, a plain text editor can be really productive.

Simple Build Tool
sbt is our Simple Build Tool, but it’s not simple or limited in any way. It’s Scala’s answer to Ant and Maven, and really it’s a superset of Ant and Maven. It’ll set up a new project, create a nice project structure for you and manage dependencies — you can slap ‘em right in by copying XML.

You can write your own build-tasks. We added support for Thrift in an afternoon; it’s got a library for shelling out, as Java is not so great at shell operations because it targets so many platforms. sbt is well well documented. And the absolutely coolest feature is that it’s got an interactive console interface where you can type in code and see how it works.

So that means sbt can insert you in an interactive way into your running program. This is great for debugging, great for sketching code out. You have a nice workflow where you don’t have to worry about compilation.

specs
We’re very test-driven, we’re not wedded to behavior-driven development (BDD), but the best library in Scala is BDD-oriented. You can throw in different mocking libraries, and it works just as well in Scala as Java.

Libraries
We’ve built a bunch of libraries. We gather a lot of stats, I mean, A LOT. We spent the first year of Twitter pushing forward on features, but never thinking about what we were building scientifically. That bit us in the ass in a big way.

You’ve probably seen a gradual increase in stability. At conferences, people ask us if it was the switch from Ruby to Scala, or if it was more machines. But really what did it was gathering numbers on everything, setting metrics and trying to improve.

Ostrich helps here. It is an in-process statistics gatherer, with counters, gauges, timers. You can share stats via JMX, JSON-over-HTTP etc. Hopefully it’s pretty simple to use and easy to integrate.

Configgy manages configuration files and logging in a really nice, flexible way. You can include config files in one another and you can do inheritance; it throws in a really nice logging wrapper, with lazy evaluation on the values you’re trying to log so you don’t burn machine-time generating log statements. It has a subscription API for pushing out a new config file. It’s a little crazy to have our own config file format, but Scala makes it work.

xrayspecs: this is an extension to specs, because we need a way to test concurrent operations. Some of the extensions in xrayspecs have been merged back into specs. We can freeze and unfreeze time.

scala-json: this is a better Scala JSON codec. We’ve used this really heavily in production for a while. If you need something like this, hopefully it’ll do the job.

Other Twitter Scala libraries: Naggatti (protocol builder for Apache Mina), Smile (Actor-powered memcached client), Querulous (a nice SQL database client) and Jackhammer (a load testing framework in its early stages). Check out GitHub for more.

How Do we Teach People?
I think we’re employing at Twitter about half the people in the world who know the Scala language. The other half are academics or at Foursquare. Even though Scala’s getting more and more popular, fundamentally we can’t hire people with experience in the language.

Pair Programming, Code Reviews
To start people out, we pair program. It isn’t mandatory at Twitter, but it’s a great way to learn Scala. We’ve come up with a bunch of style guides. The good and bad thing is that Scala’s going to be C++ in ten years, because there’s just a lot of surface area and it can get complicated. For that reason, we are pretty rigorous about a style code.

We do code reviews; it doesn’t go into the master branch if it hasn’t been reviewed by your peers. Right now, I’m working with a guy we hired from Google. He’s an amazing engineer, far better than I am, but at first he didn’t know Scala.

When I looked at his code, there was absolutely nothing wrong under the hood. But we’d go through and say, “Here’s where this line could be a little more idiomatic from a Scala perspective.” I do classes over lunch – but you need a big group to commit to come every week. Then there’s my book, and there’s other books: Dave Pollak’s book, the Odersky book (Programming in Scala, aka “the stairway book”). If you learn by example and need a desk reference, grab “the stairway book.” Or search Google for a talk by my co-worker on “The Seductions of Scala” for lots of examples

What Version of Scala Does Twitter Use?
We use 2.7. It’s got a couple of warts, particularly in the collections classes. Scala 2.8 fixes a lot of those warts, and there’s a bunch of performance work in there too, plus the ability to have named arguments in your functions.

I’m co-organizing a Scala summit at the OSCON conference in Portland this summer; come to that if you want to learn more! There’s a great blog called DailyScala, where an engineer writes about what he’s learning. I learn stuff from that guy all the time…

And that was it! Many thanks to Alex for his magnificent talk, and to all the lovely folks who visited our offices! We had a lot of fun, we learned a ton, and now we’re looking forward on May 20 to hearing from Cloudera’s Jay Hammerbacher — the man who conceived of and built the data team at Facebook — on Hadoop. Everyone’s invited!

Laziness (in proxies) is a virtue

We use Hibernate for object-relational mapping (ORM) and organize our code into domain objects and data access objects.

Historically many of our domain objects have been marked with the “@Proxy(lazy = false)” Hibernate annotation. This annotation tells Hibernate that it should NOT create lazy proxies for the annotated class.

At Redfin, these were almost all bugs. We should never use “@Proxy(lazy = false)” without a big comment explaining why it’s necessary. Our default should be “@Proxy(lazy = true)”. Laziness is good!

Here’s my quick understanding of the effects of the @Proxy annotation. As with everything in Hibernate, each individual piece seems simple, but when you consider all the features that Hibernate exposes, and how they interact, it can become pretty complicated.

Hibernate Load Options

When Hibernate loads objects that refer to other objects (i.e. have member objects), it needs to do something about the associated objects. For example, suppose that Cat objects contain (optional) references to Owner objects. When Hibernate is loading a Cat object into memory, it has to decide what to do about the Owner member variable. There are a number of things it COULD do:

1. When it constructs SQL to load the Cat, it could include the Owner table and columns in the SELECT clause, so that all the data is loaded at once
2. It could load the Cat object, and subsequently load the Owner object (via a second SQL statement)
3. It could load the Cat object, and set the Owner member to a placeholder (a proxy), which can be filled in later when the Owner information is needed

Note that it CANNOT simply do nothing about the Owner- if it instantiates a Cat and leaves the Owner member null when the DB says that the Cat DOES have an Owner, then consumers of the Cat will be misinformed- they’ll think that the Cat has no Owner, which is false.

Option 1 (get all the info in 1 SQL statement) is efficient when loading multiple Cats for which the Owner information is needed. For example, if some code needed to iterate over 1000 Cats, and get Owner information for each one, this approach would be efficient.

However, option 1 is inefficient in cases where the secondary information is not needed. E.g. if some code needed to iterate over 1000 Cats but did NOT need to get Owner information, then loading the Owner information is an obvious inefficiency.

Worse, taking option 1 to the extreme can cause an explosion in the data load. For example, a Cat might have an Owner, the Owner might have a Home, the Home might have a Address, which might have a City, which might have a State, etc. Loading the whole object graph into memory via SQL could be very inefficient. Further, every change to domain objects could cause many SQL statements to get hairier (e.g. adding a Country member to the State object would effectively add to the SQL needed to load Cat objects.)

Option 2 (load the Cat, then load the Owner) is simple, and often not bad, but never optimal (if you know you’ll need the Owner info, it’s more efficient to load it in a single SQL statement; if you know you won’t use it you should never load it; if you won’t know until later, delaying the load is better.)

However, option 2 is particularly bad when bulk operations are being performed. For instance, if some code were to load up every Cat object in the database to do some processing, this could be accomplished via a single SQL statement (though it’d probably be better to break it into chunks of, say, 10,000 Cat objects.) However, Hibernate would run a “SELECT * FROM owners” type statement for every Cat object that has an Owner- potentially millions of SQL statements.

Option 3 (load the Cat and set it’s Owner member variable to a proxy- load the Owner info on demand) is a compromise. It allows code to do bulk operations without loading the ancillary information (e.g. load all Cat objects without ever loading any Owner objects.) However, it requires additional SQL statements to load the secondary information IF that info is needed (e.g. if code loaded all Cat objects, then accessed the Owner for each Cat, option 3 would result in potentially millions of SQL statements.) Note that if the Owner information is never needed, then option 3 is most efficient- the information is never loaded.

Hibernate allows programmers to influence which strategy it will take. It offers (at least) two types of control: direct control over the SQL it generates, and control over the proxies.

See https://www.hibernate.org/315.html and https://www.hibernate.org/162.html for information on Hibernate fetching strategies and lazy loading.

Controlling SQL

When you’re implementing a DAO method, you can tell Hibernate whether it should proactively fetch information about member objects.

Under the Criteria API, Hibernate lets you call criteria.setFetchMode to tell Hibernate that it should load the additional info immediately, or should defer it. Hibernate uses the term “eager” to mean “load immediately”, and “lazy” to mean “defer loading.”

When using HQL, you can use the FETCH keyword to specify the fetch mode, which is equivalent.

When using SQL, you can use the query.addJoin method to tell Hibernate that you’ve written SQL which retrieves information for member objects. In this case, you’ll be responsible for writing the joins, etc., yourself.

Controlling Proxies

Hibernate also lets you control the existence and behavior of proxies via the tags mentioned above. Annotating a class with “@Proxy(lazy = false)” tells Hibernate to NOT support lazy proxies for that type of object (of course “@Proxy(lazy = true)” tells Hibernate to support lazy proxies.) This allows the writer of the domain object to essentially override the wishes of the writer of the DAO. If the DAO writer would like to load members in a lazy manner, but the domain object in question doesn’t support lazy loading, then Hibernate will NOT lazy load the object (since it cannot.)

If you’re writing a class for which lazy loading would be dangerous, then you SHOULD disallow lazy proxies, since DAO writers probably won’t understand the detailed load requirements of your class. However, this is unusual. In most cases, lazy proxies are safe.

Since the writer of the domain object can control what choices are available to the writer of the DAO object, they need to use that power judiciously. You CAN code all of your domain objects to disallow lazy loading, which will force all writers of DAOs to use load options 1 or 2 (load all members via fancy SQL, or load all members via secondary SQL statements.) But you generally should not. DAO writers often rely on option 3 (lazy loading), particularly when they know that the member objects will never be accessed (or when they’re not sure.) If you specify “@Proxy(lazy = false)”, you’ve made it impossible for DAO writers to use option 3, which means it may be difficult for them to get their code to perform well. Worse, the writer of the DAO may not realize that you did that, or may not understand the implications. Hibernate queries are actually kinda hard to view, so the writer of the DAO may have created a huge performance problem and not even known it (until you go into production.)

Only the client really knows

Even the writer of the DAO doesn’t know how the client will use the objects it returns. If you’re implementing the CatDAO, you might add a method like getBasementCatsAndOwners, which would return all black cats and pre-fetch the corresponding owners. You think you’re clever because you’ve avoided a major performance problem, but a caller might try to get the Home for each Owner, defeating your pre-fetching strategy. The DAO writer should do their best to anticipate the needs of their callers, and to name and document their methods such that callers can understand what they do, but ultimately the caller is in control, and can (unintentionally) defeat the optimizations of the DAO writer. If your database were large and you knew that you had clients that sometimes needed Owners, sometimes needed Owners and Homes, etc., you might make three methods: CatDAO.getBasementCats, CatDAO.getBasementCatsAndOwners, and CatDAO.getBasementCatsAndOwnersAndHomes.

Conclusion: @Proxy(lazy = false) is generally evil

As mentioned above, when you indicate that a domain object should not support lazy proxies, you make it hard for DAO writers to get their code to perform well. Worse, you disable a capability that they may be counting on, and they may not notice until there are major performance problems. Unless you have a good reason to, use “@Proxy(lazy = true)” on your domain objects.

P.S.

Lazy proxies do have some known problems.

First, the lazy proxy is NOT the same as the actual object. If you depend on the datatype of the object, you may have problems, since the type of the proxy isn’t the same as the type of the actual object (e.g. a proxy for an Owner is not actually an Owner- it’s a subclass.)

Second, you may have to think carefully about methods like equals() or hashCode(), since the proxies may not do what you expect.

P.P.S
Thanks to carloneworld for the great lazy kitty photo!

One Week After The Outage

We launched a major new version of Redfin.com a week and a half ago. The headliner was the addition of near-real-time “solds” data through our MLS-based virtual office website (VOW) data feeds. On launch day, we had a 3 hour outage and intermittent “brownouts” for another 2 days after. We wanted to give people an idea of what happened and what we’re doing to make sure this kind of outage doesn’t happen again.

Better and… Bigger
For 14 days prior to launch, we ran data imports day and night. We added 1.4 million records, 9 million photos, and revamped our internal database schema. As a result, the disk space used by our Postgres database grew by 30%. Way more disk was needed to store photos.

By Thursday morning, we were not able to go live with all our slave databases as planned. We use Slony replication for our slave databases. Errors in scripts can cause a Slony slave to require a complete re-sync, and that is, unfortunately, what happened to us. We launched believing that our single master database would handle the load. We were wrong.

By 9am PST on Thursday, our site was maxing out. First it was slow, then it was non-responsive. The problem wasn’t a rush of traffic from the press coverage. The problem was our single master database. The increase in database size and new schema overloaded it. We ended up throttling our database to allow most people to access Redfin.com, but this just caused intermittent issues and “brownouts,” where the site would be overwhelmed with requests and become non-responsive for a minute or two at a time.

Many engineers spent all Thursday and Friday looking at code, looking at the database, and looking at the traffic. Everyone was looking for some magical bug that was causing the problem. In the end, the solution was very simple. Once the slave databases were synced up and put into production on Friday at 8pm PST, the problem mostly went away. We’re still investigating the root cause, but all indicators are strongly pointing to the idea that we just didn’t have enough RAM to avoid disk I/O slowness and thrashing.

Lessons Learned
Redfin learned that the scalability & performance testing that we do before every release isn’t good enough. This outage hurt our professional pride, and we are newly dedicated to fixing this. We need to know every new release is going to run well against expected load and existing hardware.

For our next major release in December, we had been planning to upgrade our master database from Postgres 8.3 on 32GB of RAM to Postgres 8.4 on 72GB of RAM. The database servers are over two years old now. Too bad we didn’t do it sooner, but we’ve accelerated the hardware upgrade to have it ready this week. We’re also intrigued by the idea of using Fusion-IO SSDs at some point.

We also plan to spend more time looking at ways we can streamline the code to run the site more efficiently on the hardware. Hardware is relatively cheap these days, but smart engineers can often find places in the code that can be made 10x faster!

And as the site grows, we’ll also look at more scalable database solutions like partitioning or switching at least some parts to Hadoop HBase. We use Hadoop for log analysis, but it’s very promising as a high-scale query engine.

I know there are a lot of folks in technology who use Redfin. What do you think? Did we learn the right lessons?

How to Set Up Hot Code Replacement with Tomcat and Eclipse

This blog post will guide you through setting up Tomcat hot code replacement (also called hotswap debugging) in Eclipse.

• What Is “Hot Code Replace”?
• What’s the Catch?
• What About JavaRebel?
• Configuring Your Web Application in Eclipse
  1. Download Eclipse “JEE” Edition
  2. Switch to the “Java EE” Perspective
  3. Configure Your WAR Project
  4. Create a New Server
  5. Magic Setting: Disable “Auto Reloading” on Each Project in the Server
  6. Performance Tip: Adjust Memory Settings
  7. Start Your Tomcat Server in Debug Mode
• Why Disable Auto Reloading?
• Disable Auto Reloading but Enable Auto Publishing
• Finding the tmp0 Fake Tomcat Directory
• Exorcising the tmp0 Directory
• Troubleshooting: What Do I Do If I Still Can’t Get It to Work?

What Is “Hot Code Replace”?

“Hot Code Replace” (HCR) lets you make modifications to a Java class and see the effect immediately in a running JVM, without restarting your application. HCR is part of the Java Platform Debugger Architecture (JPDA) and is available on all modern JVMs.

Consider this ordinary application:

public class Sample {
  public static void main(String[] args) {
    String foo = "unchangeable";
    foo += blah();
    System.out.println(foo);
  }

  public static String blah() {
    String bar = "bar";
    bar += "blah";
    return bar;
  }

}

If you debug this class in Eclipse, you can make changes to it, on the fly, without restarting the JVM. For example, try setting a breakpoint on the second line of blah(). Next, change the literal blah to quz. Save the file and the program will continue running with the new code.

You can do this with Tomcat web applications in Eclipse, but it’s a lot trickier.

What’s the Catch?

There are a few limitations when using hot code replace. You can’t use JPDA HCR to change the signature of a class (add/remove methods or fields) or to add new classes on the fly. Additionally, some method calls (“stack frames”) can’t be modified, including the main method or any method invoked via reflection, that is, by using java.lang.reflect.Method.invoke().

Here’s what happens if you try to replace the unchangeable variable in the main method of Sample.java above.

What About JavaRebel?

JavaRebel is a hot code replacement system that’s a little better than JPDA HCR. (Maybe a lot better.)

With JavaRebel you can add/remove methods and classes without restarting Tomcat. However, JavaRebel costs $149 per developer per year, so it may or may not be worthwhile for you.

Configuring Your Web Application in Eclipse

1. Download Eclipse “JEE” edition

   Most developers already use this, since it’s the first option available on the Eclipse download page, but if you’re using “Eclipse IDE for Java Developers” (92MB), you’ll need to go back and download “Eclipse IDE for Jave EE Developers” (189MB). It’s worth it, I promise!

   

   Note: The difference between the regular Java IDE and the Java EE IDE is that the JEE edition comes with the Eclipse “Web Tools Project” (WTP), which includes “Web Server Tools” (WST). The terms are sometimes used interchangeably; keep an eye out for this if you need to search for them.

2. Switch to the “Java EE” Perspective

   Make sure you’re in the “Java EE” perspective, not the “Java” perspective. If it’s not in the upper-right corner of your Eclipse window, you might need to enable it manually (Window menu -> Open Perspective -> Other…). If “Java EE” doesn’t appear on this list, you’ve probably downloaded the wrong package of Eclipse; go back and download the Java EE version.

   

3. Configure Your WAR Project

   From scratch: From the New menu, select “Dynamic Web Project”. Configure your source and output folders, as well as your “Content directory”, which will contain your JSPs, your WEB-INF directory, etc.

   Via Maven: Use Maven to create a WAR project. For example:

   mvn archetype:create -DarchetypeArtifactId=maven-archetype-webapp -DartifactId=YOURNAMEHERE -DgroupId=YOURNAMEHERE

   Modify your pom.xml to include an explicit reference to maven-eclipse-plugin, like this:

   <!-- ... -->
   <build>
       <!-- ... -->
       <plugins>
           <!-- ... -->
           <plugin>
               <artifactId>maven-eclipse-plugin</artifactId>
               <configuration>
                   <wtpversion>2.0</wtpversion>
               </configuration>
           </plugin>
       </plugins>
   </build>
   <!-- ... -->
   

   Now generate an Eclipse project from the command line, like this:

   mvn eclipse:eclipse

   Here’s an example Maven project you can use. Just download it, extract it, and run mvn eclipse:eclipse to generate your Eclipse project. (If this is your first time using Maven with Eclipse, you’ll also need to add an M2_REPO classpath variable in your Eclipse workspace preferences that points to your Maven repository, typically $HOME/.m2/repository or %USERPROFILE%\.m2\repository.)

4. Create a New Server

   From the New menu, select Other… -> Server -> Server. For your server type, expand the “Apache” folder and select the version of Tomcat you intend to use. Choose “Next” and then specify the path to your Tomcat installation directory, e.g. c:\tools\tomcat-6.0. Add your web project as a “resource” to this server.

5. Magic Setting: Disable “Auto Reloading” on Each Project in the Server

   You now have a weird pseudo-project called “Servers” in your Project Explorer. In the explorer, your server looks like a folder called something like “Tomcat v6.0 Server at localhost-config” …but that’s not what you want. You need to interact with your server using the “Servers” tab. (Eclipse calls these tabs “Views,” but everybody else just calls them “tabs.”)

   The “Servers” tab should be exposed by default, but in case it isn’t, you can expose it via Window -> Show View -> Servers. From there you can double-click on your server to configure it.

   Note that the configuration panel for your server has two tabs, “Overview” and “Modules”, down at the bottom of the window. Click on the “Modules” tab to bring up the list of projects associated with the server.

   Select your project in the list and click on “Edit”. You’ll see the magic secret checkbox: “Auto reloading enabled”. It’s checked by default. For the love of Pete, uncheck it!

   (It’s interesting to note that JavaRebel also requires disabling auto reloading to work properly in Eclipse.)

   

6. Performance Tip: Adjust Memory Settings

   Before you start up your server, I strongly recommend adjusting your server’s -Xmx memory settings; otherwise, it will constrain itself to the default value, 64 MB, which is just not enough!

   Double-click on your server in the “Servers” tab and switch to the “Overview” tab. Click on the “Open launch configuration” link. Switch to the Arguments tab; there you can add relevant memory settings to the “VM Arguments” section. For example, you might add -Xmx512m to the end of the existing arguments.

   

7. Start Your Tomcat Server in Debug Mode

   Now you can right-click on the Server in your Servers tab and choose “Debug”. Changes you make to your JSPs or Java classes will be instantly hotswapped into your running WAR.

Why Disable Auto Reloading?

Auto reloading is a feature of Tomcat that allows you to replace Java classes at runtime without using JPDA. In this mode, Tomcat uses Java classloaders to try to unload classes and reload them; whenever it reloads, it also tries to reinitialize your application, re-launching any servlets that are marked load-on-startup in your web.xml file.

As a result, Tomcat auto reloading may not save you any time at all if your webapp has a lot of startup code. For example, if your load-on-startup code needs to warm up Hibernate database caches, Spring/Guice dependency injection configuration, etc., it may take almost as long to restart your webapp as it does to restart Tomcat.

Worse, an app that has been auto reloaded can behave strangely, and can quickly run out of PermGen memory due to frequent unloading/reloading of classes. When this happens, restarting Tomcat typically fixes the problem. If you spend even five minutes investigating a weird auto reloading problem, you’ve just wasted all the time you were hoping to save by avoiding a restart. (Not to mention your stress and frustration!)

By disabling auto reloading and using JPDA hot code replace instead, you get a more reliable code replacement system.

Disable Auto Reloading but Enable Auto Publishing

In the screenshot above you can see how to disable auto reloading on the “Modules” tab of the Tomcat server; auto reloading is bad for JPDA debugging. But there’s another setting called “Automatically publish when resources change” on the “Overview” tab of the Tomcat server. It’s hidden by default, collapsed under the “Publishing” section. You can see it if you expand that section; you want to make sure auto publishing is enabled while auto reloading is disabled.

To understand the difference between auto publishing and auto reloading, we’ll have to go into how exactly Eclipse WTP works. When you create a “Server” in Eclipse, the IDE creates a fake Tomcat directory, complete with the conf, logs, temp, webapps and work directories. When you configured the server, you told Eclipse where to find Tomcat, but it doesn’t use any of your settings files or any data from your webapps directory. Instead, Eclipse launches Tomcat with special command-line arguments, indicating where to find the fake Tomcat directory.

“Publishing” means to populate the fake Tomcat directory with all of your code. It copies your JSPs, JARs up your Java, regenerates settings files, etc.

If you turn off auto publishing, then you have to right-click on your Server and “Publish” your changes manually every time you save your Java code. Additionally, auto publishing allows Tomcat to reload JSPs automatically, regardless of whether you use JPDA or not.

Finding the tmp0 Fake Tomcat Directory

Sometimes it can be helpful to look inside the fake Tomcat directory and see what’s going on in there. Eclipse tells you where it put the Tomcat directory in the “Server Locations” section of your “Tomcat” server configuration panel. (Double-click on your Server in the “Servers” tab to open the configuration panel.) Typically, Eclipse says that your server is in .metadata/.plugins/org.eclipse.wst.server.core/tmp0; for this reason I typically call it the tmp0 directory (pronounced “tempo”).

The .metadata folder is inside your Eclipse workspace directory. (You can find your Eclipse workspace directory by going to File -> Switch Workspace; the default value is your current workspace directory.) In the worst case, you can always just search your hard drive for tmp0. It’s there somewhere!

Inside, you can see all the folders Eclipse has created. Check out the generated server.xml file in tmp0/conf. Examine generated .java files in tmp0/work. Your tmp0/webapps directory is probably empty; Eclipse has probably generated your webapp in wtpwebapps.

Exorcising the tmp0 Directory

Unfortunately, sometimes Eclipse gets a little confused about what to put in your WAR file, and you need to perform various stages of exorcism depending on how badly your tmp0 directory is messed up.

• Try republishing your tmp0 directory. Open the “Servers” tab, right-click on your server and select “Clean…” (not “Clean Tomcat Work Directory…”). Then select “Publish.” That should completely rebuild your tmp0 directory.
• Try restarting Eclipse. This works more often than I’d like to admit.
• Try completely deleting and recreating your server. Follow this ritual:
  1. Open the “Servers” tab, right-click on the server and select “Delete”.
  2. Make sure “Delete unused server configuration(s)” is checked, then click OK.
  3. Look at your “Servers” pseudo-project; make sure the folder for your server is gone. If it isn’t, right-click on it and Delete it.
  4. Quit Eclipse.
  5. Go find your tmp0 directory (if it’s still present) and delete it from your file system.
  6. Launch Eclipse and recreate your server from scratch.
• Try creating a new workspace. File -> Switch Workspaces: specify an empty directory. Create your server from scratch.

Troubleshooting: What Do I Do If I Still Can’t Get It to Work?

• My project works in regular Tomcat, but doesn’t work in Tomcat under Eclipse

  Try using Eclipse to generate a WAR file for Tomcat. Right-click on your web project and select Export -> WAR file, and install it in Tomcat by dropping the exported WAR into your Tomcat webapps directory.

  If the exported WAR file doesn’t work, then you now have two WARs: one working WAR generated by your build script, and one non-working WAR generated by Eclipse. WAR files are just zips; extract them both, find the difference, and fix it! Right-click on your web project and select “Properties”. The problem is somewhere in here.

  On the other hand, if the exported WAR file does work, then you know that the problem has to do with the way Eclipse is launching Tomcat. Find your tmp0 directory (described above) and poke around. Does everything look OK in there? Be sure to check your server.xml file, as well as your webapp itself in wtpwebapps. Make sure to note your WEB-INF/lib directory, typically in tmp0/wtpwebapps/YOURAPP/WEB-INF/lib.

• Tomcat is throwing NoClassDefFoundError or ClassNotFoundException

  First, double-check whether this problem happens in regular Tomcat. See My project works in regular Tomcat, but doesn’t work in Tomcat under Eclipse above.

  If this problem occurs in the exported WAR file under regular Tomcat, then your webapp is probably missing JARs. See My exported WAR is missing JARs below.

  If the exported WAR works but your webapp is still broken under Tomcat, you may need to perform an exorcism. (See Exorcising the tmp0 Directory above.) If this happens to you often, double-check that you haven’t accidentally disabled auto publishing. (See Disable Auto Reloading but Enable Auto Publishing above.)

• My exported WAR is missing JARs

  Right-click on your web project and select “Properties.” The problem is somewhere in here. Make sure you see your JAR listed under “Java Build Path” in the Properties dialog.

  Beware: not every JAR in “Java Build Path” gets exported to the WAR. The list of JARs for the WAR is under “Java EE Module Dependencies.” If a JAR/project is unchecked on that list, it won’t appear in your WAR file.

• My tmp0 directory is missing an important configuration file

  Eclipse will publish files that it finds in the “Tomcat” folder of the “Servers” pseudo-project to your tmp0/conf directory; if you’re missing files, you can add them here.

• My server.xml file is messed up

  That file is copied from the “Tomcat” folder in your “Servers” pseudo-project to your tmp0/conf directory. But note that the server.xml file is at least partially autogenerated by Eclipse. If you make direct changes to the file, Eclipse will do its best to try to incorporate your changes, but it often gets confused and does the wrong thing. When possible, it’s better to find the appropriate Eclipse settings and change them there instead of modifying the server.xml file directly.

  Note that one of the most common problems with server.xml is an incorrect path attribute on your webapp’s <Context> element, causing your webapp to appear on a non-standard URL. See the following question about 404 errors for more details about this problem.

• Tomcat is giving me strange 404 errors

  First, double-check whether this problem happens in regular Tomcat. (See My project works in regular Tomcat, but doesn’t work in Tomcat under Eclipse) If it happens in regular Tomcat too, then it’s probably a bug in your code.

  If the problem only happens in Eclipse, then it’s probably a server.xml configuration problem. Check your tmp0/conf/server.xml file’s <Context> element; check the path attribute. The path attribute indicates the virtual directory of your webapp. For example, if your Context/path is “examplePath”, then your webapp will appear at http://localhost:8080/examplePath. If it’s misconfigured, your webapp may not be available at the URL you expect.

  The path attribute is auto-generated based on settings in the properties of your WAR project. Right-click on your web project, select “Properties” and go to the “Web Project Settings” section. There’s only one setting here; it’s called “Context root”. Specify the context you intend to use here. If you want your project to appear in the root directory, you’ll need to put / as your context root (since you aren’t allowed to leave it blank).

  

• Tomcat times out when starting under Eclipse (“Server [...] was unable to start within 45 seconds”)

  The Eclipse developers, in their infinite wisdom, have added a timeout to Tomcat startup. If Tomcat doesn’t declare a successful startup in 45 seconds, it kills Tomcat automatically. (Gee, thanks, guys!)

  You can increase that timeout. Open the “Servers” tab and double-clicking on your server to open the server configuration panel. Make sure the panel’s “Overview” tab is selected. Expand the “Timeouts” section and increase the Start timeout to something reasonable for your server.

• I had Tomcat working under Eclipse, but now it’s broken and I can’t figure out why

  You may need to perform an exorcism. (See Exorcising the tmp0 Directory above.)

• My web project starts up fine, but when I save .java files in Eclipse, it doesn’t take effect until I restart
  • Did you make sure to launch the server in Debug mode, as opposed to Run mode? JPDA only works when you Debug the server.
  • Is your server configured to auto publish? (See Disable auto reloading but Enable auto publishing above.)
  • Did you change something that broke JPDA? (See What’s the catch? above.) If you make large changes to your classes, JPDA may be unable to replace the code; if you choose to “Continue” past that point, further changes will have no effect.
• My web project starts up fine, but when I save .jsp files in Eclipse, it doesn’t take effect until I restart

  This is typically due to disabled auto publishing. Double-check that your server is configured to auto publish. (See Disable auto reloading but Enable auto publishing above.)

  If that doesn’t work, examine your tmp0 directory to make sure Tomcat is using the correct JSP. It should automatically begin consuming new JSPs as they are installed in the tmp0/wtpwebapps directory.

• Whenever I save a .java file in Eclipse, Tomcat restarts

  This is typically due to Tomcat auto reloading. Double-check that you correctly disabled auto reloading. (See Magic Setting above.)

• Tomcat in Eclipse is much slower than regular Tomcat

  Try increasing your memory settings as described above, if you haven’t already.

  Try running Tomcat in “Run” mode (as opposed to “Debug”) mode. If that fixes the problem, then there may be nothing you can do about it. JPDA does have some overhead; you can turn it off, but while you’ve turned it off you won’t have access to hot code replacement.

---

Posted by Dan Fabulich on September 30, 2009 12:28 PM

Permalink  |  View Comments  |  del.icio.us  |  Digg