Sunday, September 29, 2013

The 10x developer is NOT a myth

Last night, I tweeted the following:


I got tons of replies and questions, but Twitter is an awful medium for discussion, so I'm writing this blog post as a follow-up.

There have been a bunch of articles that claim that 10x developer doesn't exist. The arguments against it generally fall into 3 buckets:

  1. The original 10x number came from a single study (Sackman, Erikson, and Grant (1968)) that was flawed.
  2. Productivity is a fuzzy thing that's very hard to measure, so we can't make any claims like 10x.
  3. There is a distribution of talent, but there is no way a single engineer could do the work of 10.

I disagree with all of these. Let's go through the arguments one by one. 

It's not one study

Although armchair scientists on Twitter and Hacker News love to shoot down peer-reviewed studies, the evidence in this case is fairly compelling and not limited to a single study. Allow me to quote the top reply from this related discussion on StackOverflow:

...The original study that found huge variations in individual programming productivity was conducted in the late 1960s by Sackman, Erikson, and Grant (1968). They studied professional programmers with an average of 7 years’ experience and found that the ratio of initial coding time between the best and worst programmers was about 20 to 1; the ratio of debugging times over 25 to 1; of program size 5 to 1; and of program execution speed about 10 to 1. They found no relationship between a programmer’s amount of experience and code quality or productivity.
Detailed examination of Sackman, Erickson, and Grant's findings shows some flaws in their methodology... However, even after accounting for the flaws, their data still shows more than a 10-fold difference between the best programmers and the worst.
In years since the original study, the general finding that "There are order-of-magnitude differences among programmers" has been confirmed by many other studies of professional programmers (Curtis 1981, Mills 1983, DeMarco and Lister 1985, Curtis et al. 1986, Card 1987, Boehm and Papaccio 1988, Valett and McGarry 1989, Boehm et al 2000)...
Read more here and here.

If you can't measure it, you can still reason about it

Even if you ignore the studies above and declare that "programming productivity" is hard to measure - which it is - we can still have a discussion about 10x programmers. Just because something is hard to measure doesn't mean we can't reason about it.

For example, how did you pick the programming language for your most recent project? Did you look up a study that "proved" the language was more effective than other alternatives? Personally, I don't need an experiment to prove that Ruby will be an order of magnitude more productive choice for building a website than, say, C. You could throw together some rough metrics (library availability, community support, documentation), but the reality is that most people make this sort of language decision based on intuitive reasoning and not a double blind study. And despite the lack of hard data, I'd bet that picking Ruby over C for website development turns out to be the right decision most of the time.

Of course, this isn't unique to programming: what "metric" could tell you one writer, artist, teacher, or philosopher is better than another? Merely from observing them, I can't give you a "productivity metric" that suggests Shakespeare, Nabokov, or Orwell were an order of magnitude better than the average writer, but the vast majority of people would agree that they are.

Programming is not manual labor

The biggest problem with the pushback against a 10x programmer is that some people think of programming as manual labor and programmers as assembly line workers. Some programmers are a bit better than others, but surely, a single programmer could not consistently close 10 times as many tickets as another! And a team of 10 will always outperform a single coder! Nine women can't produce a baby in 1 month!

The logic above makes it sound like programming productivity is all about typing speed; as if the 10x programmer is simply the one that produces 10 times as much code as the average guy. This line of reasoning ignores that programming is a creative profession and not manual labor: there are many, many ways of solving the same problem. Instead of the baby analogy, think more of a crime solving analogy: 10 average detectives versus one Sherlock Holmes. Who will solve the crime faster?

A 10x developer will have insights and find solutions that would never occur to an average programmer; they will avoid entire categories of problems that eat up enormous amounts of time amongst average programmers. 10 engineers writing the wrong code could definitely be out performed by a single engineer writing the right code.

Programming is about choices

Consider how many decisions go into building a single software product, such as a website: what language do you use? What web framework(s)? What do you use for data storage? What do you use for caching? Where do you host the site? How do you monitor it? How do you push new changes? How do you store the code? What kind of automated testing do you setup?

10 average programmers will make "average" quality decisions at each step and the costs or benefits of these decisions will multiply. Imagine traffic increases exponentially, and this average team setup an average website, with a data storage engine that's hard to shard, hosting that doesn't have enough redundancy, version control without proper backup, no CI environment, and no monitoring. How productive will those 10 coders be if they are spending all their time putting out fires?

A single programmer could outperform this team of 10 if the programmer can model the problem in a way where there is an order of magnitude less work to do. From years of experience, a great programmer will know that errors are much more costly to fix later. By making good decisions up front, a 10x programmer may avoid months of work down the line.

It's not about writing more code; it's about writing the right code. You become a 10x programmer not by doing an order of magnitude more work, but by making better decisions an order of magnitude more often.

This isn't to say 10x programmers make no mistakes at all; but programmers make a huge number of choices every single day and great programmers make the right choices far more often than average programmers.

And this isn't unique to programming. Would you rather have 10 average scientists or 1 Isaac Newton? 10 average scientists did not come up the laws of motion, theory of gravity, binomial series, calculus, etc; a single Isaac Newton did. Would you rather have Michael Jordan on your team or 10 average players (note: Jordan got paid ~10x the average NBA salary)? Would you rather let Steve Jobs or Elon Musk run a company or hand over the keys to 10 average entrepreneurs?

10x programmers are rare

It's important to put things into perspective. Star programmers, athletes, writers, and scientists are exceedingly rare. I wouldn't recommend building a hiring strategy around solely hiring "rock stars"; you'll end up looking foolish and lonely. Don't let perfect be the enemy of good: hire the best engineers you can get and give them ample opportunity to develop and get even better.

However, don't fall into the fallacy that all programmers are created equal. There is a vast spectrum of ability in any creative profession. On one end are the type of hires that can sink an organization, actively increasing tech debt with every line of code they write. On the other, there are people who can write code that changes what is possible and have an impact that is an order of magnitude greater than the average.



24 comments:

Frost said...

My main problem with the "10x engineer" (or "rockstar") getting thrown around is (as you mention) 10x engineers are rare, what is not rare is the amount of engineers who think they are 10x

I'm glad you brought up the issue of Jordan's pay as many times the comparison is made to sports, musicians, or Hollywood. Yes, Jordan was paid far more than the average NBA player and true "rockstars" make many orders of magnitude more than the average musician. So why are companies not paying multi-million dollar salaries to the very best engineers? (Actual salary, not options, Jordan got paid well even if the Bulls lost every game of the season)

Perhaps the pointy haired bosses of large corporations are the ones that don't buy the 10x myth. But pointy haired bosses love saving money and having one engineer do the work of ten others saves a huge amount of money, even if you have to pay him/her 10x as much because there are additional per employee costs that don't increased (e.g. you are only paying for one health insurance plan, one desk, one laptop, etc, not ten).

But even more important are those companies founded and led by former engineers who acknowledge 10x engineers exist...so why aren't they paying their engineers 10x as much?

Yevgeniy Brikman said...

@Frost: great question.

Salaries for non-athletes aren't typically public information, so I can't be sure, but I'm guessing some distinguished engineers, CTO's, VPs, and the like make *significantly* more money than the average engineer. The salary is likely not 10x, but with equity, it could be. And no, I don't think stock options can be ignored here; different professions pay money in different ways; also, sports teams give huge bonuses based on individual and team performance too.

However, I suspect there are also many cases where a 10x engineer doesn't get 10x the compensation. This is probably due to one of two factors:

1. Management doesn't recognize or believe in 10x engineers. They see software as an assembly line and believe every programmer is a roughly equivalent assembly line worker. This is the kind of thing I'm arguing against in this post.

2. The contribution of a star engineer is not always as obvious - or as impactful - as a star athlete. Jordan was 1 of 5 players on the court (yes, there are subs, but Jordan would play almost the entire game), and every time he played, the team won more games and far more people watched. A star engineer may be 1 of 20, 200, or 2,000 engineers in a company and the success of the company depends on not just the software, but also marketing, distribution, brand, etc. Star athletes, actors, and musicians are much more in the spotlight - even if there is a team behind them - and can demand much higher salaries. Not sure what we can do about this in software.

jaideep said...

I don't know what point articles like this proves. When developers can do really terrible job, the difference can become 100x or 1000x or even more ! The point is you can't possible generalize things like that because every programmer works with different set of conditions (like different environment, languages, work).

So the fact that some programmer can work 10x better than someone else doesn't really prove anything or help anyone. You can only just keep learning from your own mistakes and learn to take the best from people. Knowing that you are 10x better than someone or someone is 10x better than you won't really help you.

Clifford Campbell said...

I've found that my productivity can vary depending on the project we are working on.

Greenfield project with clear requirements using the technology of my choosing? I am going to be way more productive in that environment then using an older technology with vague requirements and a ton of legacy restraints.

In general though I do agree that '10x developers' can exist. I know some really smart and productive people in this industry.

TeslaNick said...

There's a disconnect between your sources and the way you frame your essay. From the source:

"However, even after accounting for the flaws, their data still shows more than a 10-fold difference between the best programmers and the worst."

And an example of misusing this information:

"A 10x developer will have insights and find solutions that would never occur to an average programmer; they will avoid entire categories of problems that eat up enormous amounts of time amongst average programmers."

Unless we know what the distribution of best-to-worst programmers is, we can't speak to the average level of productivity, we can only compare best and worst. Let's say Bugatti Veyron is 10x better than a Chinese Cherry QQ. That says nothing about how my Subaru WRX stacks up to either of them; it's somewhere in the middle of the bell curve.

If we assume a bell curve distribution, the best programmer would be about twice as productive as the average, not 10 times.

Adam Smith said...

I don't dispute that there can be a big difference, but I don't think that difference is constant across every type of project. Some programming jobs require less thinking than others.

The problem with claiming there are developers who are 10x more productive, is that managers might think that they can just find these developers and then their projects will finnish in 1/10th the time.

This does not take into consideration how well a team works together. With shitty organization even talented people can't produce much. It also doesn't take into account that gathering requirements, talking to customers, getting feedback etc can't be done 10x as fast.

While I don't doubt that there are rockstar developers I think it is not trivial to use them efficiently in an organization.

Yevgeniy Brikman said...

@TeslaNick: good point. The studies found differences from ~5x to ~30x between the "best" and "worst" programmers. Of course, "worst" and "best" is relative to the folks at the actual study, so it's possible that when you really get the outliers in there, the variation would be even larger. This suggests there is an order of magnitude (or more) variation amongst programmers - that is, there is evidence for "10x" programmers - but it isn't enough to say much about how the average programmer compares to the best. Thanks for bringing it up.

Unknown said...

Are there 10x programmers? Yes, but only when compared to someone that can't program, doesn't have the skill for the task, or someone so lacking in the subject matter of the assignment.

Basically I'm saying that 10x is a relative term, no an absolute, but I loved comment that there are a lot of programmers that think they're 10x!

It reminds me of a great quote...

"The difference between genius and stupidity is that genius has its limits." - Albert Einstein

Ogrface said...

10x engineers are rare, but in the ones that do exist, what are the patterns in how they became 10x? Is it just years and years of experience? Is it the environment they came up in? Did they all read the same books or stick with the same technologies throughout their careers? How does an "average" developer put in place personal goals that will make them into a "10x" developer?

Steve Webb said...

I've worked with a few people who seem to be viewed this way by management. Unfortunately the people at the coal face usually look at the code and realise yes it was produced quick, yes it does seem to work but yes its brittle as hell, is built on mud and not solid foundations. My view is there are a lot of smart idiots in the world.

Information Technology said...
This comment has been removed by a blog administrator.
Information Technology said...
This comment has been removed by a blog administrator.
Noah Gibbs said...

In response to salary versus developer productivity in the comments:

Different engineers have different aptitudes. I absolutely believe that there are engineers who could produce, say, a Hadoop-based reporting system for a given web site's analytics 10 times (or 50 times!) faster than an average programmer with basic training, and do a better job.

I believe that person has not produced 10x the business value of the first programmer. Mostly they have produced about the same business value with 1/10th the costs, which is a lot less than 10x the business value. So they make more, but not 10x more, and that's appropriate.

There *are* programmers who create 10x (or 50x!) the business value. But "doing something similar, a bit better and much faster" does not directly create that much business value.

Only a programmer in a more executive function -- one who can choose *what* problems to solve -- will actually produce 10x the business value. And those guys *do* make 10x the salary, as they should. But their title doesn't usually include "programmer" or "software engineer" even if they're building the system... Or more commonly, directing a team of other engineers in how to build it.

IParrot Post said...
This comment has been removed by a blog administrator.
IParrot Post said...
This comment has been removed by a blog administrator.
aman said...
This comment has been removed by a blog administrator.
pjb said...

Ok, I'll react on this assertion:

"but surely, a single programmer could not consistently close 10 times as many tickets as another!"


I agree.

What a 10x programmer could do, or any reasonably competent programmer, if allowed to, is to close 0 tickets while the others would close n ticket/day. He would think about it. Why are there tickets to be closed? If they have to be closed, is how do close them all at once? But better, what to do to just avoid having them in the first place.

Let assume that most of those tickets are bugs. Perhaps the problem is that of legacy code and technological debts. Probably, the best solution would be to just re-generate a new, bug-free program, from the specifications.

But then, what specifications? How often have you seen specifications written before the spaghetti code, if at all?

Perhaps being able to close all the tickets at once would involve a major "refactoring" (what an euphemism!) of the existing code. How often are the programmers allowed to perform such refactorings?


pjb said...
This comment has been removed by the author.
Alan said...

My similar views on the same subject are here - http://angryweasel.com/blog/?p=726.

Thanks for writing this.

Nancy jaee said...
This comment has been removed by a blog administrator.
Jain Technosoft said...
This comment has been removed by a blog administrator.
Kent Kemmish said...

Okay, this might seem a bit mercenary, but hear me out, and please help.

I'm not a recruiter, I'm a scientist. But, still, I've got to find people, and in the past, I've done it better than any recruiter I've known- so I keep doing it while my project requires it.

I'm working on a stealth, well-funded biotech startup that is doing something ridiculously cool and even historical. I've thrown all my heart, soul, and mind into it, and I had huge opportunity costs to join.

So, please read this carefully and think- you could make a historic impact by just introducing me to the right people.

I need *genuine* 10X coders- the real superstars- the ones who are contacted by recruiters constantly- to get on the phone with me. Forget every bad interaction they might have ever had with a recruiter- they need to hear about this project if they have any passion for technology or for the idea of hacking biology and saving our lives. I am a scientist and alpha geek myself, and I find phone calls with extremely talented people almost always go well, and that even if they're not interested personally, their imagination is set on fire by the project, and they'll think of *other* 10X coders I should talk to, *or* they'll find a home with our project in the longer run.

I need to meet or talk to those rare folks. I already know several that I think are 10X, and have hired some. Yes, we've had "Michael Jordan" level engineers and scientists join the team already- and I want to find more. We've had people leave their dream jobs and their own companies. With my previous company I helped found, which shares some of the technical team and ideals with this one, we even hired a tenured professor full-time, someone at the very top of a worldwide field.

I want anyone who is *genuinely* a rockstar- not, as some posters mentioned, just someone who thinks they are- to reach out to me. I want to talk to as many of these 10X engineers as you can send my way. I think such people in general will *resent* not getting the chance to hear about this project. It is the kind of thing you kick yourself for years for not hearing about sooner. For someone who wants to code stuff that will *really* have a major impact on the health of this planet, please contact me. I'm Kent Kemmish, you can email me at robots4synbio@gmail.com for more information.

T J said...

Here is Gabe Newell discussing his most productive developer... this guy seems to blow the 10x developer out of the water.

http://www.youtube.com/watch?v=Td_PGkfIdIQ#t=696

Alan Abraham said...
This comment has been removed by a blog administrator.