Jeff Hammerbacher on Hadoop, Facebook and a Surprising Bit About Microsoft

As part of our engineer-to-engineer series of technical talks, Cloudera co-founder Jeff “The Hammer” Hammerbacher visited Redfin’s San Francisco office last week to explain why Facebook began using Hadoop — which stores data as a lot of big files on many computers rather than squeezing it into the rows and columns of a database — for analyzing how people use Facebook. In case you forgot, Jeff led the team that built Facebook’s data storage systems. Now, Redfin engineer Gordon Brown just posted a summary of Jeff’s talk. My favorite bits:

Software Developers as Business Analysts: Jeff emphasized the importance of letting developers access usage data, so they can figure out on their own what optimizations need to be made to the site. While the query language of traditional relational databases doesn’t do much beyond pull the data out, Hadoop allows developers to build their own analytical tools, or extend those of others, using a more powerful all-around programming language like Java.

The Rise of the Machines: Jeff observed that the amount of data being stored by computers is exploding, just because most of it is now automatically captured by machines. You can change one setting on a web server and increase the amount of data you capture about what people do on your website by a factor of 100 or even 1,000. When Jeff worked at Facebook, the company was generating a terabyte of data — a trillion bits of information — every day. And this was in 2007.

Microsoft Has Started to Do a Lot Right: While Jeff was hardly impressed by the packaged software bought by businesses less enterprising than Facebook — Gordon was too polite to include Jeff’s unexpurgated opinions — Jeff reserved special praise for all the analytical tools Microsoft offers with its database, SQL Server. As Jeff said, “It’s kind of scary that Microsoft has started to do a lot right within the last 5 years.” If you do end up working with a relational database rather than Hadoop, Microsoft’s a good choice.

To read the entire summary of Jeff’s talk, visit our devblog. Or check out other engineer-to-engineer talks, such as Twitter talking about its use of Scala. The next big talk is our own Sasha Aickin, dishing it out next Thursday on the merits of HTML5 vs. proprietary mobile applications. Everyone’s invited! And many, many thanks to Jeff for a fantastic talk.

Discussion

  • http://pc.de/fonts/ Island Eye

    Time for me to get to work …. thanks for sharing your experience.

  • Pingback: The Rise of the Quants | Redfin Corporate Blog

  • http://www.air-jordan-18.com air jordan 18

    The post about leisure is quiet good. Reduce our work pressure, improve our life mood . I like it very much.I will support it often. These days I want to buy something from these websites but I dont know how to do.I hope you can help me.Thank you very much! adidas arenas Signature

  • Aleman

    Javea is built round a large bay between the headlands of San Antoni and Cap de la Nao. Most of the holiday activity in Javea is centred around the Arenal beach, which is also where you will find most of the holiday villas. This “bay within a bay” is surrounded by bars, cafes, restaurants and shops to keep you entertained throughout your holiday.

  • spanish1

    Javea also boast a large British ex-pat community, which help to bolster its economic standing and ensure that there are always going to be other driving forces in the area to increase economic stability.

  • http://www.javea-estates.com/ long term rentals Javea

    The prime places worth staying at in Javea can be summarized into three areas. There is the very much clean beach

    area, the port and the homey village. Tourists who want some action can take a look and a get a feel of the many

    recreation areas like the bars, and restaurants that surround the harbor area. Apart from these, the quaint shops

    are all around the village area. Tourists do not come home to their native places without checking out these

    stores for souvenirs and great bargains. This only proves that there are a lot to do and see at Javea.

  • http://www.jps.im/ villa holidays javea

    The coastline from here becomes more picturesque than ever and if you are driving, you'll want to keep your camera at hand. It's considered inside the district of Valencia. Examining some of these incredible vacation spots more discreetly, each villa is considered to be someone's favorite, as it has something extraordinary that will tickle your fancy

  • http://www.jps.im/ villa holidays javea

    The region attracts visitors looking for plenty of outdoor activities in a mild sunny climate. The Jalon Valley

    is particularly popular with walkers, cyclists and artists who come to paint pictures of the amazing

    countryside, vineyards and orchards. The region also offers excellent horse riding, fishing, golf and

    watersports on the coast. Walkers can enjoy the trails in the spectacular Barranco del Inferno Gorge, the

    amazing open ridge walks in the unspoilt limestone mountains of the Marina Alta, visit the quaint traditional

    villages of Lliber, Guadalest, Orba, Alcalali and Parcent, and take a welcome break at the beauty spot of Algar

    Falls. Nature lovers will love the abundance of flora and fauna at Montgo Park near Javea and the Pego

    marshlands. For the more adventurous who love an adrenal rush there is paragliding and rock climbing in the Coll

    de Rates Mountains