Service Oriented Architecture with Varnish and Edge Side Includes

As we talked about before, Redfin uses Varnish to implement Edge Side Includes (ESI.) This involved breaking a single big (and expensive) page into individual chunks; each chunk would be generated by separate code, and would be cached on a different schedule.

Once we broke our expensive page into chunks that could be individually cached, it seemed pretty easy to have those chunks served up by different backend servers. Voilà, a monolithic app became “service oriented“! This would let us run the different software components on different machines (with different performance characteristics, different SLAs, even implementations in different languages/environments!)

Of course, nothing is actually that easy, and we made a number of mis-steps before we figured out how to do it.

How To

Varnish allows you to define multiple backends in your VCL. And in your vcl_recv function, you can decide which backend should handle a particular request. At Redfin, we added a new Varnish backend for each of our ESI endpoints, and we added logic to choose the relevant backend by URI. In practice, we actually only have one pool of machines handling our ESI requests, so all of our Varnish backends actually point to the same place.

So the first piece of the puzzle is on our main web servers. On the main web servers, requests go through Varnish. Requests for “normal” pages are sent through to Tomcat, but requests for ESIs are sent to one of the SOA backends. Here’s an example of what the VCL file might look like:


backend default {
  .host = "localhost";
  .port = "8080";
}
backend similars {
  .host = "similars.redfin.com";
  .port = "6081";
}
backend relevantlinks {
  .host = "relevantlinks.redfin.com";
  .port = "6081";
}

...

sub vcl_recv {
  if (req.url ~ "^/esi-listing-similars" || req.url ~ "^/esi-property-similars") {
    set req.backend = similars;
  }
  else if (req.url ~ "^/esi-listing-trackbacks") {
    set req.backend = relevantlinks;
  }

You might have noticed that the “localhost” backend is associated with port 8080 (where Tomcat is running), but the ESI backends are associated with port 6081 (where Varnish is running on those remote machines.)

We want the instance of Varnish on the main web server to cache content from the main web server, and the instances of Varnish on the ESI backends to cache the content from those backends. This has a few benefits:

  • Our effective cache is bigger, since we have caches on multiple machines, each of which has fixed memory
  • Having independent caches prevents one set of items from pushing another set out of the cache. If all the data were in a single cache, then cache entries holding similars information (which is small, but expensive to recreate) could be pushed out of the cache by cache entries of “main page” content (which is big and relatively cheap to recreate, but we’d still like to cache.)
  • It’s easy to flush individual caches without having to worry about performance problems with other parts of the site

We have another design goal: we’d like to have a single distribution of our software. We’d like to have a single WAR that we can put on any machine; we do NOT want to have to deal with multiple builds, with figuring out which build has been installed on which machine, etc. We’d like to be able to switch a single machine from being a standard web server to being an ESI endpoint without having to redeploy or reconfigure.

This creates a conundrum. We want our main web servers and our ESI servers to be identical, but we also want them to act different. In particular, when an instance of Varnish on a web server gets a request for an ESI fragment, it should redirect that request to an ESI server (more precisely: to the Varnish instance running on an ESI server.) But when an instance of Varnish on an ESI server gets a request for an ESI fragment, it should forward the request to the local Tomcat instance. It should NOT forward the request to ITSELF. Forwarding port 6081 to port 6081 creates an infinite loop- not good.

We want to break the symmetry between the standard web servers and the ESI servers, and we do that by messing with the URIs.

We prepend our ESI URIs with a known prefix, which means “forward this to the ESI server.” But when we process the URI (while forwarding it), we strip off that prefix, so that the ESI server does not also forward it to itself. That’s harder to say than it is to code. The VCL code looks like this:


sub vcl_recv {
  if (req.url ~ "^/backend/") {
    set req.url = regsub(req.url, "^/backend/", "/");

    if (req.url ~ "^/esi-listing-similars" || req.url ~ "^/esi-property-similars") {
      set req.backend = similars;
    }
    else if (req.url ~ "^/esi-listing-trackbacks") {
      set req.backend = relevantlinks;
    }

This breaks the circularity. The path of requests looks like:

  1. A requests comes into Varnish on the standard web server for /path/to/a/page
  2. Varnish forwards the request to the local Tomcat instance
  3. Tomcat responds with HTML that includes <esi:include src=”/backend/esi-listing-similars” />
  4. Varnish processes the ESI, and must make a request for /backend/esi-listing-similars
  5. The Varnish instance on the standard web server strips off “/backend”, and sends a request for “/esi-listing-similars” to the ESI server
  6. The Varnish instance on the ESI server gets the request for “/esi-listing-similars”
  7. Since there’s no “/backend” prefix, the Varnish instance on the ESI server forwards the request to its local Tomcat instance
  8. The Tomcat instance on the ESI server processes the request, and responds with the relevant HTML fragment
  9. The Varnish instance on the ESI server caches the HTML fragment and returns it
  10. The Varnish instance on the standard web server parses the HTML fragment into the main page content and returns it to the browser

This example points out another tricky bit- how do we assure that the HTML fragment is cached by the Varnish service on the ESI server, but not by the Varnish service on the standard web server? To handle this correctly, we add a header to the response which indicates if it’s already been cached:


sub vcl_fetch {
  if (req.url ~ "^/esi-") {
    if (obj.http.X-RF-Cached ~ "true") {
      pass;
    }
    set obj.http.X-RF-Cached = "true";

This code says “If there’s an X-RF-Cached header present, then don’t attempt to cache. If there is NOT an X-RF-Cached header present, then add one, and attempt to cache.” With this addition, the HTML fragments will only be cached on the first Varnish instance they pass through, which is on the ESI server in our case.

How NOT To

The solution described above works, and meets our requirements. But we also tried some solutions that did NOT work. Perhaps you can learn from our failures…

Putting Absolute URIs into ESI Includes

Our first thought was that we’d put absolute URIs into our ESI includes in the HTML. For instance, we tried to put <esi:include src=”http://similars.redfin.com:6081/esi-listing-similars” /> into the main HTML of our page. Varnish simply (and correctly, I think) ignores the host name and port. Including http://similars.redfin.com:6081/esi-listing-similars will cause Varnish to act as if you included /esi-listing-similars, and Varnish will use whichever backend it thinks is relevant, regardless of the host name or port in the URI.

Using a Single Server as both a Standard Web Server and an ESI Server

When doing testing, or when some of our servers were unavailable, we were tempted to use a single server as both the standard web server and the ESI server. It seemed like this should work- the trick with the “/backend” prefix should prevent infinite circularity. However, it didn’t work. It seems that Varnish is doing its own checks for circularity, and noticing that a single request passed through the same Varnish instance multiple times (which NORMALLY would be a problematic example of circularity, but we’ve got our clever symmetry breaker in there!) Anyway, Varnish doesn’t allow it, and causes those semi-circular requests to fail.

P.S.

Thanks to D’Arcy Norman for the photo!

Discussion

  • http://twitter.com/thekaanon Kaanon MacFarlane

    Good stuff!

  • karatedog

    My problem is that breaking a huge html into chunks works if and only they are served by the same application, and where those chunks don't carry any relevant information to the main page. In short, ESI cannot work as a loose application server.
    If any of the chunks is a 3rd party app, or a different application that should give back useful data in its response header, that information will be lost to the client, because by the time this chunk is “executed” Varnish has already transmitted the HTTP response headers back to the client.
    I don't see too much trouble waiting all the chunks to be loaded and THEN do the response (ie. with all the different headers packed into the response) I don't know why this is not an option in ESI…