Does It Really Move the Needle?

One increasingly common refrain I hear among product managers, designers and engineers everywhere is the question of whether a new project will “really move the needle.” During an idle chat with Redfin about how we could analyze traffic patterns and touring activity to determine if a listing was likely to sell quickly, the great Dan Fabulich declared he didn’t want to talk about it since it was hard to imagine if it would “add millions of new unique visitors” or “raise revenue 20%.”

I love Dan, precisely because I have never met such a pure engineer — he wears Ms. Pac Man t-shirts quoting Dickens and runs Harry Potter conferences — with a deadly commercial instinct.  His comment brought to mind a drug dealer I once heard about, who happily declared on an FBI tape that “if it ain’t a CONSIDERABLE piece of business, I ain’t gonna f*** with it!” As my brother’s law-school class would later learn, this statement was enough to move the dealer up from petty crime to racketeering. In our own world as dealers of addictive websites, we all want to think like gangstas, and think big.

But there’s also a place for artistry and craftsmanship, for starting small. When painting the Mona Lisa, Leonardo da Vinci didn’t ask himself if each brushstroke would really move the needle. He was painting first to please himself, alone and ecstatic. The most profitable websites I’ve ever seen were made by people who weren’t obsessed with profits. They wanted to fix a problem that drove them nuts or to impress a cute guy or to create something gorgeous for the sheer joy of it. They weren’t afraid of failure, and they didn’t “pivot” when faced with their first setback.

But now, as the bubble drives us all to think like Groupon or Twitter, we are in a year of frantic, magical thinking. We lose sight of the user, and lose sight of ourselves, in pursuit of the almighty dollar. The beginning of the end for Microsoft’s innovative phase was when it began quashing projects that couldn’t conceivably lead to billion-dollar businesses, as if anyone could really know how tall a flower can grow before watering it. Redfin began as a website designed to please home-buyers and home-owners — and designed first to please ourselves — from which, quite improbably, tens of millions of dollars have flowed.

I am not arguing against a healthy dose of skepticism for any website feature, nor for the quarter-bath, search-by-houseboat incrementalism that leads to a complicated user experience rather than a big new idea. I love it that Dan is thinking big, and thinking deadly, about whether a new project can become a killer application. But when we think about a killer application, we still want to think first about whether it will delight people, starting with ourselves. From that, the money will usually come.


  • Dan Fabulich

    For the record, I don't RUN Harry Potter conferences; I'm just one of many volunteers!

    • GlennKelman

      Don't ruin my perfect image of you Dan!

  • rechangeagnt

    You could not be more spot on,  (I commented on your post  5/26) your teams focus on that type of negotiation power will help you grow your consumer confidence base.  I hope you know the power your team has in this multi-billion dollar industry, which is experiencing unprecedented paradigm shifts from an industry perspective, and from a consumer perspective.  You are the future of real estate! I can say that coming from experience.  Keep your engineers hyper focused on the consumer and 20% revenue growth is just the start.

    • GlennKelman

      That's very kind of you to say! Some days we feel that way, some days we don't.

  • Venturion

    Yahoo, Google, Craigslist were powerful and simple web sites from inception. Great businesses start from one core invaluable offering.

    • GlennKelman

      Venturion! Good to see you on the blog. Google has kept getting better though, hasn't it?

  • Stcloud Realtor

    I recently came from St
    Cloud Realtor
    blog and have been reading along. I thought I would
    leave my first comment. I don't know what to say except that I have enjoyed
    reading. Nice blog. I will keep visiting this blog very often from <a href="”

  • ignoblenoble

    are you kidding? all we talk about in the redfin forum is whether prices will go up or down, and whether a particular house is over priced or under priced.

    nobody today provides predictions on whether a listing is over priced or under priced. if you guys did it well, you'd be the first. and it would be an awesome feature.

    it's true the best websites, delight the user in unexpected ways, they don't try to squeeze every cent out of them. engineers often think very linearly. it's okay to take a detour once in a while.

  • A.

    Thank you for beautifully summarizing the other side here.  Your points really hit home for me.  I've heard them before in various guises, but your metaphor about how tall a flower will grow, really sums it up.  Paraphrasing:  (1)  initially design for an audience of one — yourself, and (2) “it's hard to make predictions, especially about the future”, an overly cited Yogi Berra quote.

    I've long been skeptical, and slightly resentful, of “metric-itis” or “dysfunctional metrics”– the dogmatic drive to only deal with decisions that can be quantified and measured.  “If it can't be measured, it doesn't matter” is the boldest declaration of this mindset.  Countered by Einstein's quote, “Not everything that counts can be counted, and not everything that can be counted, counts”.  Don't get me wrong, I'm an avid engineer and love data, but only the intelligent use of data.

    I wish more people in the business community applied a finer level of critical thinking to their analysis, instead of reverting to a “silver bullet” mindset in which “metrics” is the One and Only True Way forward.  More often than not, their proclaimed motive of truth-seeking usually falls subservient to one of forcing an agenda or outcome, e.g. “how can we make this data make ourselves look good?”

    I always rattle off four quick counterpoints:
    (1) correlation isn't causation — just because traffic spiked in response to your feature launch, doesn't necessarily mean it was the cause.
    (2) metrics have an economic cost to set up, audit, and verify, that is almost always dismissed with “of course we always want MORE metrics”
    (3) it's easy to commit statistical fallacies, e.g. taking (and comparing) the average of averages, is a fallacy, unless the ordinalities match.
    (4) most importantly, rarely do I see consideration given to how a metric will alter performance.  This is the “performing to the metric” fallacy, also known as “managing for optics”.  Most commonly cited example is a customer service center that tied employee bonuses to their time spent on each phone call.  The result:  employees started hanging up on customers in mid-sentence at the 10 minute mark (to maintain their avg calls per hour velocity metric).

    • GlennKelman

      Great comment A., really thoughtful post.

  • Ben Elowitz

    This synthesizes a struggle I've had continually in product design. 

    On the one hand, I think the best products in the world are far more than the sum of their parts.  An iPhone is not a mathematical equation adding a cell phone + a GPS chip + a touch screen + accelerometers + a great display + a very good UI.  If Apple products are magic, it's because Apple gets every so many details right – the bounce on the scrolling all the way to the visual effects when you click the home button or toss a picture into an email.  None of those minor effects — NONE of them — are needle movers.  And while Apple is the easiest case to point to to show it, it's true of other great products too. 

    But it's something that most cultures lack.  Intuit's Quicken started with magic decades ago, and has all of the soft wins beat out of it by hyperrational needle moving.  Microsoft's Office products seem to have their souls ripped out of them like an inflamed appendix, as though the calculus deems that souls don't give you features on a bullet list or better usability test results, much less a rationale to choose Microsoft over Google, so they must be discarded.

    Myself, I've gone both ways on this, and I wish I could say that I've evolved to know when to pick one over the other.  How did the founders at YouTube know that there's magic in easy uploading, transcoding, a global marketplace of video… but that the playback quality just wasn't that important in 2005?  How does Steve Jobs know that the iPhone doesn't need to be very good for voice calls, as long as it's the ultimate personal lifestyle device?

    My business career has grown my left-brain like a tumor, swallowing up the right.  I swear I have a whole lobe now dedicated to looking for needle-movers; I think it's where rhythm and dancing would have been if I hadn't gotten an engineering degree.  If it's not growing a key target by 10% then don't bother, goes the reasoning.  A startup's time is its scarcest resource – even more scarce than money.  So scarce, that it can't wait for the rationality of adding up all the pieces.  It needs to go to making wild, crazy, right-brained bets – doesn't it?  Some of the worst product decisions I've presided over have been when I did the needle-moving calculus.  And yet, most of the best avoidances of waste have been when I stopped to ask the question:  will this really move the needle? 

    The question I want to know the answer to is this:  for the most successful, for the Steve Jobs' and Glenn Kelman's of the world, how do you know when to use the calculus, and when to just make a bet to go for magic, calculus be damned?

  • Pingback: Eric Nehrlich, Unrepentant Generalist || Encouraging useful failure || November || 2011