Under the Hood at Twitter

Twitter’s Alex Payne visited our San Francisco office yesterday as part of an engineer-to-engineer series of talks on new technologies and best practices for building large-scale consumer web applications. We just posted a recap of Alex’s magnificent talk on our developers’ blog.

Alex is an interesting person. He has a tattoo of a Chinese character on his left arm to remind him that change is constant. He was the second or third engineer at Twitter, joining in 2007 to build out the programming interface that introduced real-time to the rest of the Web, and now he is one of the people leading the charge to keep the service up and running at terrifying speed and scale. He has strong opinions about San Francisco, a city he left for good last night. He was driving several cats up to Portland, but had never heard of Mitt Romney’s preferred method of animal car transport.

What I’ve always found most curious about Twitter is that the service is innovative not just in what it does, but in how it does it: despite facing more reliability and scalability pressure than any modern online service, Twitter has embraced a somewhat obscure programming language called Scala to solve its unique problems. This seemed like a big, scary bet (offset by the fact that Scala can use Java libraries and tools). As Alex discussed in his talk, Twitter today employs as many as half of the world’s Scala programmers.

What emerged in the discussion was a fascinating exploration not only of Scala, but also of how any new company has to approach building a large-scale web service: the decisions it makes, the tools it builds, the architecture that it evolves. We invited folks from all over the San Francisco developer community, and many came — from as far away as Apple. By the end of it, we were all deeply impressed at the self-confidence, openness and thoughtfulness of Twitter’s engineers.

Thanks to Alex for coming by on what must have been a hectic day, to the folks at Greylock for their help in putting this together with Digg and Pandora, and to everyone who visited us. You’re all invited to Jeff Hammerbacher’s talk on Hadoop, which should be equally good, since Jeff conceived of and built the data team at Facebook. For now, check out the summary of Alex’s Scala talk; it’s interesting even for folks who have never written a line of code to understand how one programmer thinks about a very big problem.

Discussion