As part of Redfin’s brown-bag lunch series, the great Neil Roseman visited us today, wearing a goofy t-shirt he bought on Woot and a sports jacket. He led Amazon’s earliest engineering efforts — he bought the company’s third web server, ever — and rose to its greatest heights. Dozens of folks from Seattle’s startup community also came by, and lots of folks listened in to the online presentation. And now all of them are emailing and calling and coming by to say how wonderful Neil was.
Neil’s talk was about how an engineering organization can stay agile while the company grows fast. It was funny, humble, insightful, goofy, incredibly compelling. Here’s what he had to say:
The fish rots from the head down: there’s a Turkish website that tries to prove every Greek proverb was originally Turkish and so we should say that there’s a Turkish saying that the fish rots from the head down. For an engineering organization to be great, you need a leader who is great. “Jeff Bezos is present, Neil said, even when he is not present.”
Seduction is better than coercion: let people choose programming languages. People become zealots about programming languages but it’s a real mistake to force people to think your way. Seduce people instead. At Evri, Neil bought a five-person software team with an enterprise background. They weren’t interested in scrum, XP, or agile programming. They came from a different world.
Neil liked putting post-it notes on a board everyone can see, moving notes along to track progress. The team wanted to build its own system. Neil said fine, so long as it’s transparent: “I don’t want to bug people to know what’s going on.” The system the team built was so complex that they switched to his system of sticky notes. They came around without feeling it was jammed down their throat.
It’s ok to be different: a single solution is rarely what is required. You’re better off letting people do what they want. If they want to deploy via Maven, let them do that. As Neil said, “a good engineering team won’t use tools that suck.” To keep the culture you had when you first started out with four engineers, let tools, processes, designs vary. If you build software correctly, your interfaces have to be in agreement, not implementations. Amazon iterated many times on how it built and deployed software; it deployed via human beings and Perl scripts, via third-party software, via different teams using their own approaches, and finally built tools that were so good that everybody wanted to use them. If you want to use your own system, you’re stuck with managing and maintaining it top to bottom.
Hiring gets harder and harder. The temptation when you get bigger is to let professionals — HR and recruiters — handle hiring. It has to be a significant part of everyone’s job. We started to worry at Amazon that our hiring bar wasn’t as high as it was in the past. Good people like to work with good people. Most managers don’t know how to make poor performers better. Managers aren’t often good engineers, and engineers aren’t good managers. Amazon got worried about the quality of the engineering team as it grew and got the VPs together to make sure that someone who understood the culture and its standards was always on every interview loop. Neil himself did campus visits; it’s never too early in a company’s life-cycle to start campus recruiting. Amazon made the time to recruit, from the highest level on down. “I’ve had SD 1’s talk to Jeff,” Neil said, “And Jeff would always make the time.”
Every one of Jeff Bezos’s shareholder letters includes this phrase: “Setting the bar high in our approach to hiring has been, and will continue to be, the single most important element of Amazon.com’s success.” It is probably why Amazon has prospered when so many other technology companies started in the 1990’s have declined. Neil also cited a recruiting letter that Jeff Bezos distributed to folks he knew; my favorite line was: “Expect talented, motivated, intense and interesting co-workers.”
Loose-coupling is really important to make sure you have the right culture; it’s a good architectural model as well. You can’t have a very large, uniform group, as you end up developing your own micro-climates anyway. Metrics gives you the latitude to let groups run however they like. UPS delivers on time more frequently than FedEx but FedEx has the better reputation because it was the pioneer in tracking information. FedEx made promises that people could validate for themselves.
Focus on small, frequent successes: it’s important to have lots of things that you can promise and then deliver on; small teams like doing that. Let a small team build a plan with frequent successes. The most demoralizing thing is to go a long time without a success. Small bits — frequent repeatable success — this is really important.
Hierarchical communication kills engineering organizations: try to avoid ever having a hierarchical org chart. The left hand doesn’t talk to the right. Hierarchical communication decreases satisfaction of employees, and requires more coordinators. There’s a lot of weirdness in humans so you have to have managers. But if you could write a Perl script to eliminate managers you would. At Redfin, if you have more software engineers and more agents, you’d make more money; that’s not the case for somebody who coordinates stuff, they’re only there because they have to be. Flat organizations have managers who aren’t’ very involved, and the team figures things out on their own. “These managers are like the scrum-masters of scrum-masters.”
Create a culture that self-selects: a strong culture lets people know whether they fit in. Let people who don’t like it opt out. “If there isn’t a clear signal for people to opt out, you don’t have a culture.” You don’t have to be nice, but you don’t have to super-aggressive either; Neil doesn’t like being adversarial, asking how many man-hole covers there are in Seattle isn’t his style. The culture works people in and out of the organization in a way more subtle than that.
Continuous is better than batch — and launch dates are batchey. Some people need a launch date. If they do, maybe they’re not right for the culture. “My attitude was,” Neil said. “it’s web software, it’s running all the time. Our software wasn’t mission-critical. When Amazon launched in Germany, we couldn’t take money for three weeks. Since this was our early days, that wasn’t such a big deal. But if the 787 fails — Boeing has 10,000 100-page specs — people die. At Amazon, if we make a mistake, nobody dies.”
Let people go: as companies change, folks get upset when people leave their team. “When somebody leaves the team you get ticked off at them,” Neil said. “I have worked very hard not to get ticked off. I say, ‘I love you, I’ll keep a seat here, come back when you’re ready to come back.'” Neil noticed that someone in one group was a star, but as soon as he left that group, the manager concludes he wasn’t so good after all. There’s an old Yiddish saying: ‘someone else’s backside is easier to smack.’ (Neil quoted the original Yiddish!) There are lots of yo-yos at Amazon, people who left and came back, because managers took a more mature approach.
Focus on the inputs, not just the outputs. Neil talked to a hotel company that measured itself by looking at its bookings. But he asked, “Why not look at how long it takes to make a booking, how easy it is, how good the experience is?” It’s like focusing on the stock price of a company. Let people have ownership over the stuff they can change on a daily basis. Teams will focus on the outputs if left alone. (This made me want to give every Redfin team its own metric.)
Measure everything. In the developer’s view of Amazon’s site, the bottom of every page has every service call listed out and how long it took to run, to focus everyone on performance. Every measurement gets better, because that’s how people keep score of whether they’re doing well. Pick the measurement carefully; and what the heck, measure everything too.
Work backwards, starting with the customer, the press release, the user manual, the mockup. This works great for consumer software but you can do it for anything. The subscription API for Amazon Prime began with a press release that Neil wrote, even though no press release was issued when it was finished, even though it had no user interface at all.
Be innovative, from the top and the bottom. Give awards. Amazon had a just-do-it award, which is given only to people who didn’t ask for permission (I love this). You need a culture where there is a willingness to fail, and to fail fast. Encourage people from the bottom up to make decisions in real time. You want a continuous flow process, not a big planning cycle. I hadn’t realized that different teams at Amazon own different elements of a single page, but Neil emphasized this; everything was chopped up so a two-pizza team — a team small enough to be fed by two pizzas — could build it.
At Amazon, people enter ideas through the ideatool, for submitting an idea directly to executives. “A single engineer put an idea in an internal tool for a shipping club and then we launched Amazon Prime in three or four weeks using a subscription API we’d already built. We launched Prime very quickly.”
Be prepared to throw stuff away. Refactoring projects should always scare the crap out of you. Refactoring rarely accelerates development. Make somebody prove that the next three releases will be 20% to 30% faster. “Most of these projects,” Neil says, “are polishing the turd.” It also isn’t so bad when you have applications that do the same thing, though engineers often complain about this; the user experience should be consistent but otherwise you need to stay loose. Infrastructure should be shared.
Don’t be afraid to let necessity be the mother of invention: let people design systems when they need them. In 2001, mobile was supposed to be huge but it didn’t work out so well. Amazon built a huge mobile team then took it down to four engineers. The engineers were about to be dispersed throughout the company; they wanted to stay together and Neil asked what can you do? They wanted to wrap Amazon in a thin layer of code to deliver a set of services as XML over HTTP and they had three ideas for how this could help Amazon:
1. Voice-based customer self-service through TellMe; this cut 90,000 phone contacts per quarter.
2. “A music thing that didn’t work at all.”
3. i-mode in Japan, which turned out to be a huge success
They stayed, and became the web services team that got Amazon into the web services business.
It should be fun. Before he began to worry that it created some potential religious conflicts, Neil used to ask people if they considered themselves lucky. “People identify themselves as lucky or unlucky; the lucky ones don’t complain, they find the good stuff to do and get it done. The other people say they would have shipped but marketing screwed them…” That said, there are a few Eeyores who are amazing performers. “My favorite Eeyore answered a question by slapping his palms on the desk, farting and leaving the room.”
Build for the biggest scale from the beginning, Neil said, and with that he was done with his presentation.
Neil then answered a few questions:
iPad vs. Kindle, what’s your take?
We created a market. Here you have a guy, Steve Jobs, who hates books deciding he has to make something that competes with Kindle.
How do you give engineers latitude and still create a consistent user experience?
To make user interfaces consistent, make as few rules as possible so people actually care about them.
How do you get engineers to work with business folks?
I’ve always wanted to have people come in to work, have a sign on the door that tells them to go to a new office, and whatever office they go to, that’s their new job for a few weeks.
How do you get software engineers to wear pagers and help out with operations?
I think it’s dangerous to have people write software be separate from how it’s used in real environments. You don’t want to create a team of lesser technical beings to maintain it. But you can write software that can be managed by people who aren’t software developers. By the time we were done, IT could manage 70% of the tickets.
How do you allocate time to experiments that can be sketchy?
Experimentation was built into the very core of Amazon. Jeff always encouraged us not to worry about things that performed less well than what was there because performance could improve and decay over time; you have to compare the performance of one thing at the starting point to another thing at its starting point.
Afterwards, Neil was mobbed, and he said one more interesting thing that I can’t help but repeat: that he doesn’t like the term software architect because it pre-supposes a rigidity in software when what is so wonderful about it is that it is very plastic, something people write for ourselves and can change how we like…
Many, many thanks to Neil for a great talk, and to all the folks in the startup community who came by to hear him speak. If you have any questions, leave a comment below and we’ll make sure it finds its way to Neil!
Menu
United States
Canada