Hacker News new | past | comments | ask | show | jobs | submit login
Let a thousand flowers bloom, then rip 999 of them out (gigamonkeys.com)
231 points by logic on Sept 29, 2015 | hide | past | favorite | 122 comments



>Engineers’ effectiveness, on the other hand, is hard to measure. We don’t even really know what makes people productive; thus we talk about 10x engineers as though that’s a thing when even the studies that lead to the notion of a 10x engineer pointed more strongly to the notion of a 10x office.

Management has a vested interest in pushing the meme that there are no 10xers: it gives them an excuse for not paying these engineers what they're actually worth. The rest of us just go along with it because it assuages the inadequacy we naturally feel when we compare ourselves with someone like Linus Torvalds or John Carmack.

Clearly there are 10xers, and some of them even post here: Walter Bright, Bryan Cantrill, and Ted Unangst.

Edit:

I'm also not surprised Seibel would say something like this given his past demonstration of lacking social awareness by titling his book of programmer interviews "Coders at Work"--a title that one of his interviewees, L. Peter Deutsch, rightly objected to, much to Seibel's shock. Deutsch in 1984 co-authored the seminal paper on building efficient virtual machines for dynamic languages through just-in-time compilation and inline caching, techniques which ultimately gave rise to the JVM. He deserves an immense amount of respect for this and other accomplishments, and we do him and our profession a great disservice by publicly referring to him as a "coder" or anything else that elicits something less than awe from the general public.

Could you imagine doctors referring to some highly-regarded brain surgeon as a "cutter" or "butcher?" No, instead they talk them up as "brilliant" or even god-like, yet we do the equivalent with "coder" (and allow the media to do it too) all the time.


Albert Einstein and Richard Feynman were 10x physicists. Then there are probably a couple more 9x physicists, many of them with Nobel prizes under their belt. A lot more 5x physicists. A whole mass of 2x physicists. Another whole mass of 0.5x physicists. I don't think anyone wants to seriously dispute that there's a bell curve of aptitude in pretty much any human endeavor, whether that aptitude came naturally or was nurtured due to hard work or both.

But the 10x gospel we see in software engineering is an entirely different beast: the claim is that there's a small group of people who "really get it" and that everyone else "just isn't made of the right stuff". It assumes both a kind of determinism (you do not become good, you either are good or you're not) and a hard dichotomy (there's no ordinary engineers, just bad ones and great ones).

Of course management would prefer to make ordinary engineers better than to let a couple of brilliant minds work their magic. But that's just common sense: https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Me...


Totally agree.

One thing I've noticed: the people who talk about 10x engineers almost uniformly believe they're either apart of that elite cadre, somewhat close to it, or will one day ascend to its ranks.

I believe that there are nodes in a network which have greater inherent value than other nodes, but can only achieve 10x value when working with other nodes of similar caliber in the context of the problem they are solving.


There are certainly 10xers. But how many 1xers are you willing to piss off to keep one of them happy? 10? Maybe you're breaking even, maybe you're even still ahead. 20? Starting to become more of a problem.

Torvalds and Carmack is an interesting twosome to start with because from where I'm sitting, having never met either of them, Carmack is a dream engineer from every angle - and coworkers alike. If I worked with him and found out he made 10x my salary and got 10x the leeway I'd shrug, nod, and say well yeah, of course.

On the other hand, if I worked with Torvalds at a company that wasn't his encompassing baby, I think I would quit in a year as would many others. He can only get away with the shit he pulls because he is a singular genius. But most people who act like him are not, yet because some managers think they're 5x or 10x or whatever, they get indulged and coddled.

I've worked with enough assholes that I'll take a couple less productive human beings over a ninja-pirate-superstar prick.


The best 10xers are the ones that make the surrounding 5 people into 2xers.


Who in management do you see pushing the meme that there are no 10xers? It's management who is responsible for that meme. It's designed to get people to work 80 hour weeks, to shun work/life balance.


The 10xer isn't a 10xer because they work 80-hour weeks but because they produce relatively more output of a relatively higher quality in a given time frame, by as much as an order of magnitude, or at least that's how I understand it.

For example, Linus Torvalds wrote git in a month and in the process effectively dug the grave of svn, a project into which thousands of man-hours had already been sunk.

The "10xers don't exist" meme supports the notion that programmers are interchangeable cogs and justifies the ~$130k salary ceiling we run into without going into management.


I think you're mistaking how easy it is to write a program from scratch when you have nothing in your way (once you have some experience, before that writing a program from scratch is really hard).

Maintenance becomes ever more complicated and hard as time goes on because the complexity of the bugs goes up by orders of magnitude as does the complexity of the code.

I've spent more time adding a very minor bit of functionality to a system I wrote 5 years ago than I did writing the whole thing in the first place. Because it grows and adding new code or modifying existing code becomes more and more expensive.

How many man-hours have gone in to git now? And it's not as if git is that great anyway, we all know how bad the UX is for example and all the man hours that have gone in to masking that problem, it's just better than SVN.


Linus could have chosen to work on improving SVN instead, but he did not. The decision to write a new version control system in the first place is part of what makes makes him so much more effective, not just pure programming ability.


One of these days I'm going to have to write an essay on my experiences as and among very smart people, and my experience of burnout. Although it's the sort of subject that attracts controversy and people saying that their personal experience overrules your personal experience.

Provisional title: "I used to be a 10x programmer like you, until I took an arrow to the knee"

(also, there are multiple different types of 10x programmer who are not necessarily cross-substitutable)


Such a claim really reeks of elitism and predeterminism. Everybody has similar inherent capacities as long as you're not born with some serious disorder. it's just that some people work harder, plan better and are more obsessed with certain ("success" or fame for example) that others don't necessarily do etc., so they appear to be more efficient at certain things. Even then still they're just maybe 50% better than other coworkers. "10x" is a ridiculously huge number and of course if you compare top programmers with college freshmen then maybe you'll get this amount but if such a discrepancy exists in a professional company then probably it hired some extremely shitty programmer and the management should thoroughly reflect upon what went wrong. That'd be the only explanation. It's like if in a professional soccer club's first team the best player is literally "10x" better than the average players then the club must have some serious issues and the top player won't stay for long anyways before he goes off to a team that makes more sense.


Just an observation. There are perhaps 10xers in 'software development productivity' but a company should pay based on a different metric, 'business value productivity'.

There is nothing saying that a 100% increase in software productivity will cause a 100% increase in business value. Sometimes the relationship will be 1/1 sometimes 1/10 but sometimes 1/0.1 or even. 1/-x. It depends on so many things (how clear and adequate the goals are, the state of development, the context of the business).


Managers are supposed to get all that money in exchange of making sure that relationship is always near 1/1.


> even the studies that lead to the notion of a 10x engineer pointed more strongly to the notion of a 10x office

Does anyone have a citation for this?


I heard that result from reading Peopleware, although this (apparently well researched) Stackexchange answer disagrees:

http://programmers.stackexchange.com/questions/179616/a-good...



If there are true 10x-ers, they'd be pretty rare, right? Like the chances of being one would be on a par with playing in the NFL.


It guess but it depends a bit on how you define 10x. If you do it by business value added by some programming it could be as simple as someone saying don't do that do this at the right point.


Nice article, but I can't agree with everything inside.

1. Engineers are not computers, so saving 5 minutes a day will not increase productivity by 1%. Most engineers work with passion, and don't look at the clock. They are focused on delivering a certain task, and they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day.

2. The benefit of the tools is a clear win, but as these tools will not change so much over time, after the initial boost of productivity, the effect will no longer be noticeable. You can't increase performance each month compared to the last month just by using the right tools. So I think the EE team make sense to be created "on demand" and not as a permanent solution. Time should be a parameter in the effectiveness model

3. If you grow to a large scale enterprise, as the people are not computers and reaching agreement between large number of individuals is not easy, the bigger the number of people in the EE team the harder will be to approach solutions and test them.


> they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day.

I'd say giving your engineers 10 minutes more free time at the end of the day will give long-term returns too. Engineers are not computers, they need time to unwind after work like everyone else.


You're missing the point entirely.


creshal has a different point. That does not mean that he/she missed the other point.


Saving 5 minutes of BS a day can increase passion which might be worth far more than a 5%.


Yet saving 5 minutes of FS (F for Fun) can decrease passion, causing a net negative effect.

In other words, maybe you're both right and it's not about saving time per-se, but about cutting BS.


As organizations grow, it can also become harder to maintain the passion you mention. This is a huge factor that people usually don't account for in their models.


    after the initial boost of productivity, the effect will 
    no longer be noticeable
But the purpose of the team isn't just to get the initial boost but also to prevent a continual loss of productivity as the tooling inevitably bitrots. When you hit a certain size that bitrot happens much faster than you might think. It makes sense to keep an EE team on just to do maintenance. And support future changes to workflow. New languages and frameworks. Large scale refactorings happen more and more frequently as you hit walls that weren't there a month ago.


Argument #1 does not really make sense, since by repeatedly applying the same logic it implies that engineers will work an infinite number of minutes each day. A variant of this argument that I've often heard is this:

"My decision to take the plane does not negatively impact the environment, since the plane would fly with or without me in it."


The point I was trying to make is that for engineers you can't do a simple math from minutes to productivity. The interruptions are big productivity killers, but all engineers take voluntary breaks. A break helps productivity, but there is a limit, above that it harms. The same with the interruptions, some might help, too many will not.


A break the engineer chooses is chosen and is unlikely to interrupt flow. An interruption caused by the tooling is not chosen and is more likely to interrupt flow.

Tooling cruft also increases the startup cost of some tasks. Where the amount of pain you will experience trying to do it may inhibit your desire to dig in.

The author was generalizing he wasn't assuming perfect accuracy. But in an engineering organization the size of Twitter the productivity gains of a dedicated tooling team add up fast. Much faster than you might think.


I could see people seriously agreeing with the plane argument.

I think to spin it even worse, replace it with some sort of group crime. I joined the riot, because there would be a riot with or without me. I join the drive by shooting, because there would be a drive by shooting with or without me.

I think that would have less people agreeing with it than the plane argument.


> I think the EE team make sense to be created "on demand" and not as a permanent solution

For the size and intensity of effort going on at Twitter, I respectfully disagree.

Once the team gets large enough, NMP ("Not My Problem" mindset) comes into play, and unless you have the culture from the beginning to always be making the development process better, and acquisitions and new management has never changed that, which is highly unlikely, then having a dedicated person or team to help make other developers jobs easier is a great idea- after the team gets to a certain size. What that size is depends on what is being done, though.


> Most engineers work with passion, and don't look at the clock. They are focused on delivering a certain task, and they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day.

Right...

First, IME engineers aren't any more passionate about their jobs than anybody else. Even for people who are passionate, the majority of their day is unlikely to be 100% dedicated to the things they're passionate about. Nobody I've ever met is passionate about digging through their inbox in the morning, or going to scrum meetings, or doing sprint planning, etc.

Second, engineers are also the ones who get most upset and demotivated by the pointless waste of time. If you can speed up my compile by 5 minutes, then do it. If you can speed it up by 5 seconds, then do it. There's nothing the engineers I know hate more than wasting time on tasks that could be automated or made to run faster.


I think the team could very well exist permanently. It's just that their aim would then not be to unify tools, but to ensure code quality and work efficiency of various teams, as the author already stated in the article. Quality assurance and checking can be beneficial all along the way, as long as it's not overdone.


Saving 5 minutes a day every day for a year will increase productivity more than 1%.


Oh I look forward to your paper detailing the studies and data that led to these completely objective conclusions.


I mostly like this post, but there's something that troubles me about it.

One of the distinctions I learned from Lean literature was that bureaucracies fall on a spectrum between controlling and supportive. I had honestly only ever experienced the former: rules and mandates to make me do what other people valued. But I came to realize that there are other possibilities. For example, when my team jointly makes a checklist we follow, that's a kind of supportive bureaucracy. I could imagine scaling that up, but it's certainly rare.

In this post is a hunger for power that makes me uncomfortable. Sure, any short-term mess can be solved by bossing people around. But I've seen places where centralized quality groups become strongly counterproductive. Every mandate they issue is intended to clean up a mess. But as developers become disempowered, they do less and less cleaning on their own.

Hopefully that won't happen at Twitter; there's a theme of wanting to build trust. But there's also the theme of ripping stuff out, of controlling the chaos, of whipping things into shape. That sounds much better to the person holding the whip than to those getting whipped with it.


    GOOOOAAAAALLLL! Fail Whale.
    GOOOOAAAAALLLL! Fail Whale.
Oh yeah, fail whale. I guess Twitter really has come a long way in the past five years. Despite all of the shit they get for not building out their product in a visually-engaging fashion, I have to say that I can't remember the last time their product failed me. Props.


That's true, but I can't help but feel like it was an extremely low bar to hurdle. If "35.6 million tweets during the Brazil/Germany final" is to be believed, that is only 5k tweets per second on average...maybe 50k peak due to heterogeneous volume during a sporting event. Call me uninformed if that's what it is, but that seems like something that should be easily handled by any decently architected system on the JVM. I couldn't even count the number of shit systems at Amazon that process that kind of volume with less than 4 9's of availability.



Facebook is getting 43x the QPS of twitter, and they are doing it on MySQL.

http://highscalability.com/blog/2010/11/4/facebook-at-13-mil...


You're quoting a post which is 5 years old and comparing it with one that's two years old.


> Once your engineering org gets to be a certain size the benefits you can obtain by investing in making all your engineers slightly more productive start to swamp the slight gains that one team might get from doing things their own, slightly different way.

I like to think of this idea as group vs. individual velocity. Yes, you personally might be able to go faster by introducing new technologies - but everyone else is slowed down by having to learn it. Knowing which few of these technologies will be an overall win is the tricky bit.


This article repeats the assertion that there is no such thing as a 10x engineer but doesn't site anything that shows such. It has become a popular meme but I know counterexamples from people I have worked with that were significantly more effective (10x minimum) engineers than others.

Does anyone have anything that shows they don't exist? I'm starting to think it's something average engineers like to repeat to convince themselves that there isn't anyone really 'better' than them. Sort of a reincarnation of the condescending "there are no winners or losers" thing we do with childrens sports.


I've come to dislike the idea of the "10x engineer" partly because it puts software development too much in the realm of factory work or even craft-work, which it is not. Fundamentally software creation is creative and inventive. It is not a matter of stamping out a certain number of widgets per day. It is not a matter of churning through some number of bugs, lines of code, function points, or what-have-you. It is design work, it is creation, it is art and artisanry. To purport to measure developers against one another along some linear measure is to miss the point entirely. It is the same as asking whether Mark Twain and Walt Whitman are "10x writers", or whether fillet mignon is 10x better than a PB&J. Not only are these things non-linear, they exhibit complexity that is irreducible to such simple measures, which is why we try to avoid the attempt (although somehow we refuse to learn that lesson when it comes to movies, television, and video games, but I digress).

The top tier of developers are far more than 10x more valuable than the average developer. Not because they produce 10x more lines of code, or "crush" 10x as many bugs or sprint points, but because they build. better. systems. Period. They write better code. They build the right stuff. They promote better team cohesion. And they execute on difficult projects more effective.

And they don't just make one order of magnitude difference, often they make multiple orders of magnitude difference.

Here are three fairly common scenarios involving such "keystone" developers:

There's a large and crucial code base which is owned and maintained by an entire team, but there is one developer who has a much different relationship to the code than the other people on the team. They wrote most of the original code, though it has grown much since then, and they know far more of it than any other single person on the team. Tasks related to the code which would require much more time and often the collaboration of multiple team members can be done alone and with much less time by this person. If they leave the project will live on, and it may even flourish, but the development velocity and the capabilities of the team will be significantly diminished compared to when the key developer was around.

There's a core feature on a very important product that is in development, and there's one developer who seems to have an outsized impact on it. They've gone the extra mile to make sure the feature is designed and implemented well, they've put in the hard work to get the feature shipped with excellent quality (both in the code behind the feature and in the way it works and feels to end-users). They've made sure that the right work has gotten done and the most important core aspects have shipped. And the end result is a phenomenal product that attracts and retains users, engenders customer loyalty, and ultimately results in a massive amount of revenue for the company.

There's a tiny team or a startup working on an ambitious product. Despite their limited resources they produce something of significant value and quality, something that normally would require a much vaster number of average developers working for much longer to achieve a similar result. This leads to runaway success.

These scenarios happen all the time in the industry, and they speak to a gulf in capabilities between the most capable devs and the industry average. And it's not necessarily about "productivity" it's about getting the right work done, done well, and done efficiently. You put a top tier developer and a middle of the road developer on the same task for a week or two and the difference isn't that one has written a lot more code than the other, the difference is that one developer has something that works and something that you wouldn't mind keeping around.

This is another thing. Good developers learn and build from their past projects. Which means that over time the developer is getting better, and also what they're able to deliver based solely on their own work grows exponentially, because it builds and builds and builds, like compound interest. Whereas mediocre developers are building foundations of sand, they can only build so high before everything comes crashing back down and they have to tear down and replace what they've built before with something a little bit better. Mediocre developers tend to spend their time running in place.

P.S. Another huge factor is culture, mentoring, leadership by example, and work environment. Top tier developers raise everyone around them to a higher level through their example and their interactions with others. Learning from better developers is a hugely important way that devs get better at their job. And merely getting into a mindset that getting better is possible and valuable is important in its own right. Additionally, working with talented engineers, people we look up to, people we take inspiration from, etc. are important reasons why devs show up to work every day. When an "anchor" dev with a lot of talent leaves a team or a company it can trigger an avalanche of talent leaving the org as other talented engineers decide that the work environment isn't as worthwhile as it once was and they leave as well, accelerating the process until a lot of the talent has evaporated from the org. This is a very common process in the industry because the people who tend to be the most talented and the most intrinsically motivated tend to have the easiest time finding other jobs and tend to be the most sensitive to the quality of the work environment. Unless you're immersed in the work environment and sensitive to these things you may not pick up how drastic these changes can be until years down the road when the impact of a team that has lost all its talent finds itself vastly less able to execute on important projects effectively.

And while I've talked about a lot of "soft" subjects such as "art" and "work environment", these things absolutely have an impact on the bottom line.


> [...] it puts software development too much in the realm of factory work or even craft-work, which it is not. Fundamentally software creation is creative and inventive.

I assure you, it is craft. It's much less in the realms of art, science, and invention, and much more in the realms of putting well-known patterns together to build something, sometimes even something new.

I assume you have never seen a modern woodworker at work?


Craft and art exist on a continuum, there is no break between them. Often people who are not or do not know artists imagine that the creation of art entails some magical skill, some special step that is lacking in craft work or engineering work. That some "quintessence" is injected into the process which breathes a vitality of art into the work that would otherwise be absent. But this is a false impression of art. the creation of art is precisely as you say, putting together well-known patterns to build something, sometimes something new.

Wood-working using the skills of working the lathe, the chisel, the rasp is fundamentally the same sort of thing as painting using brush strokes. And both skills can be used to create either a work of art or a work of craftsmanship. It's a matter of skill and intent. Fundamentally though, creating a painting, a poem, or a song is mostly craftsmanship, it's the application of skills to build something in a familiar way.

There's no magic breath of life that elevates a crafted work into the realm of art, rather it's the purpose it's made for and other factors which make that distinction. Something that is considered "not art" would typically be something that is completely unoriginal and has a purely practical purpose.

Art, on the other hand, tends to involve creativity and originality, it tends to not just be for a practical purpose but also to be evocative, emotional, and communicative.

You see, the method and skills of construction are not the differentiator here. Both a EULA and a sonnet can use the same medium (written language) yet while one is purely practical and ordinary the other communicates more than mere facts and details, involves some degree of originality and creativity, and engages the reader at a higher level.

Software dev is a very wide ranging field, so it is difficult to talk about it holistically. However, in my experience it does not fit the description of unoriginal rote work. Often there is creativity and originality, as well as subtlety and fine artisanship. It is a new kind of thing which does not neatly fall into either the manufacturing/engineering/"craft" or "art" buckets, and for that reason it requires different ways of thinking about the work and the workers.

The most important point here is that when you give craftsmen the task of making 20 chairs, they will do so. Some with greater efficiency than others. But this task is nonsensical when it comes to software, because it is only ever necessary to make one of a thing in software, copies are free (or as nearly so as makes no practical difference). Instead the equivalent task would be to make 20 different chair designs. And now it becomes more clear that we are not dealing with craftsmanship, we are dealing with design work. Creative work. The closest analog to which is art. And how do you judge an order of magnitude difference in design work? How do you objectively measure the difference between an Eames lounge and plastic patio furniture? Objectively you can't, but subjectively we are aware that the difference between middle of the road design and top tier design is much more than a single order of magnitude.


> They write better code. They build the right stuff. They promote better team cohesion. And they execute on difficult projects more effective.

Let's put it this way: 10X engineers ship 10X better products. Some places still use the daft metrics you mentioned (LOC, bugs squashed, etc.), sure. I'd assume that a gig such as Twitter would know that a 10x engineer is identified by, essentially, gut feel and can't be quantified. There seems to be two definitions of the meme and I'd assume that most people around here would refer to the more realistic one (10X better products).

More facts of the 10X meme: not every engineer is necessarily 10X on every project you might assign them and, as you mentioned, a 1X hire could end up being a 10X engineer some day.

Back to the article. An EE team isn't the only way to approach this. We've got at least two where I work:

With enough 10Xs around sometimes you just need to let them fly: give them a little freedom to fix shit, take their fixes seriously and trust them. There's a good chance that if something annoys an engineer enough, they will take their own initiative and fix the shit (sometimes in their own time). This is the approach that my team lead takes - he'll give us the time once we can prove the value of our changes.

Another team lead I chat to a lot logs tech debt the very instant that it is created and allocates time every release to remove tech debt. His tech debt backlog is rapidly approaching zero. The advantage here is that his junior engineers learn the cost of tech debt and learn how to make the correct decisions to fix it as they are involved with removing it. An EE team with 10X elites provides no knowledge sharing.

An EE team might not work everywhere (especially for us: our product could be seen as a collection of products), the most important take-away from that article is that tech debt can be solved with a little responsibility and by granting your employees enough autonomy to figure out how to solve it.


excellent writeup!

what I found is, that while we are engineers we like technical solutions to our problems. the problem is, that writing software is a team effort where social skills are required to produce good results- a non technical skillset we do net hear very much about at university...


Engineering is a social activity. Engineering degree programs do attempt to impart non-technical skills required for success in the field. This is further evidence that software engineering needs to be turned into a bonafide engineering degree program.


Though this is true. No one knows everything and the chain is as strong as its weakest link. Ultimately 10x programmers have to end up working with 1x 2x 3x.. you get the idea. They also have to deal with tools and bad decision making and bureaucracy and real life. Success is often defined by how well everyone and the market come together. Was Jack a 10x programmer, I doubt that very much. The focus should be on how to make everyone better. Not how each programmer can show off his skills at the cost to other people who might not be as good as him. In fact that's why programmers in the Valley aren't making a million a piece. They don't stick together, they compete to outdo each other, and management moves in for the kill.


Oh, I didn't intend to underplay the importance of having an effective team. It just feels like we are pretending that extremely skilled people don't exist and I'm not sure why. I know a principal engineer at (giant networking company) that does clear close to 1 million a year after bonuses and stock because of the special project he is on and he was invited to that project because he is so effective.


The concept I think is completely flawed. Many people are extremely productive at some point in their career but usually tend towards average. I think its highly dependent on their work, level of interest in it, team support, personal situation etc. Any of those change and that 10 just evaporates.


That's not really the case for any other skill. There are going to be people far on the right side of the distribution and they will be a lot better regardless of those factors.


Amusingly, that was Chairman Mao's approach as well.


Yeah, from the title you'd expect an article about Chinese politics. Turns out it's just an Effectiveness Engineer using fancy phrases without understanding their historical significance.


This happens so often in this community that it's cringeworthy.

When I got to the first mention of Twitter I hit the back button.


And people say the humanities aren't important.


At first I expected an article on Twitter platform for 3rd parties where 1000 flowers bloom and 999 of them get ripped.

Twitter has a horrible, slow moving, bloated interface. I m facing countless examples of disturbances every day.

Maybe a single platform based on PHP could be better, if Facebook is so usable?


Really, Facebook?

Yesterday I spent half an hour on my mobile just to find out where the "view profile as it would look to someone else" page had been moved.


I strongly suspect the mathematical formula suggesting that 10,000 engineers could be as productive as 45,000 with the right support is a case of "in theory, but not reality."

When tree farming was invented, the initial crop was wonderful, but they soon had to come up with a term meaning "forest death" for the sickly, underdeveloped trees that followed. A monoculture of the same plants quickly strips the soil of essential nutrients and if you cannot identify those nutrients and aggressively resupply them, you will soon find that the trees you wanted to grow and harvest will no longer thrive and somnetimes will no longer survive. A forest can produce healthy trees for generations because of the diversity of plants growing therein. Different plants use different nutrients and some replenish the nutrients used by neighboring plants.

Obviously, software tools are not plants. But I have my suspicions that the diversity of code he describes developed in part for reasons like "different engineers happened to be experienced with different languages, tools, etc" but there may be cases in there where it developed that way because it was the best way to handle it. So when you start standardizing things, you may be killing off things that are critical to the health of the codebase and the company.

I can see value in having someone dedicated to serving the needs of the engineers so they can be more productive. I can see a need to clean things up. But complexity itself often has inherent value that cannot be replaced or improved upon with a simpler solution. Sometimes, simplifying things amounts to throwing the baby out with the bath water. I hope they don't wind up doing that.

https://en.m.wikipedia.org/wiki/Forest_dieback


That was an enjoyable read. As mentioned in the article, here's the link to the same talk he gave at Facebook's @scale [0], though the sound gets ridiculously out of sync after a bit.

[0]https://www.youtube.com/watch?v=8IyXcLFO9ns


> Alex Roetter, now our SVP of Engineering, then an engineer, personally led the effort to convert what Scala code the ads team had already written into Java.

I'm having trouble understanding why on earth you would rewrite Scala code to Java. The Scala code runs on the JVM and can interoperate with the Java, so what is the point?


That's cool if the Scala bits are mostly static bit, not so much if they see active development.

When (say) 5% of your codebase is Scala and the rest is Java, and your engineering team prefers working in Java, those 5% become stumbling blocks. The context switching slows you down and increases the risk of introducing errors.

Also, a tool chain that will support two languages in parallel, while obviously not impossible, is necessarily more complex than only supporting one.


Maintenance and when adding more features to it.

Obviously runs and interops fine, but for less paper cuts when they are modifying it or near to it, it makes sense to refactor it to a common base.

And removing the scala plugin from poms will make the project(s) simpler.

Though as a Scala dev, not the direction I would have chosen, ever.


I'm sure faster compile times are at least part of it.


> Twitter’s first line was written in 2006 by our now interim CEO, Jack Dorsey.

Really? Not Noah Glass?

Also, no mention of Pivotal Labs role in helping Twitter get rid of the fail whale...


What role did they have? Unless you mean the pivots that Twitter hired full time? I worked at Twitter during the entire transition from the monorail to the current architecture and there was ~0 Pivotal Labs involvement in that process. I know they had a big impact on the earlier phase of Twitter but when it comes to getting rid of the fail whale a large part of the credit goes to Marius and Raffi, neither one of whom were Pivots and I can't think of a single Pivot that built a service at Twitter that is in production today.


I'm really interested in the "standardization" part near the end. Is the end goal to have Twitter building everything with Java, or everything with Scala, or something like that? It does seem to go against the grain of "use the best tool for the job".


> E = (eng - ee) * (1 + b * ee^s )

He posits that model and draws many conclusions from it. But how reasonable is? I would expect the improvements in efficiency factor to decay much faster. If everything is done exactly right, there is an optimal efficiency factor. With no ee's there are deviations from that perfect process. As more ee's are added, more deviations are fixed and you get closer to optimal efficiency. These assumptions would mean that efficiency should converge as number of ee's approach infinity. My model suggestion would be

E = (eng - ee) * (1 - b * s^(-ee) )


The article discusses the kinds of problems that wildly successful teams of 1000-s of engineers have. As such it is kind of irrelevant for most of us - not everybody will grow their team to 1000 engineers or assume the role of VP Eng in a large organization. A more practical matter to consider, are these problems worth worrying about from the start or is "let the thousand flowers bloom" strategy part of the twitter's success?


I was discussing this article with another ex-Twitter employee yesterday. I quit in the middle of this monorepo clusterfuck, and my views are very different - I believe most of what happened was a Battle of the Egos. I can name names but its not worth the trouble, Twitter was a generous employer etc. He remembers most of this as a Battle of the Languages ie. Scala vs Java vs Ruby thing. Seibel has this 3rd version, having joined much later in 2013. The list of ex-Twitter engineers is probably in the thousands at this point, so you are going to hear a whole different set of narratives.Twitter is Roshomon 2.0. One thing is certain. The lack of profitability is very much due to throwing lots of cash at lots of engineers who wrote and rewrote the same bits until kingdom come, driving each other and the market nuts as to what all the fuss was about.


Once you get 4.000 Engineers in EE, it is worth considering creating some tools specifically for EE. Let's call this team Effective and Efficient Engineering, thus EEE.


The question that has been arising for me, is what amount of resources do you devote to engineering effectiveness in a relatively small engineering org (~25 people)?

It seems like the presented formula would say to not devote any resources, but there seems to be some good return on some effort put towards it. Right now, I've just considered it some percentage of my job to work on.


If you have only 25 people, you have few enough that you can reasonably discuss just about everything with the whole team. So, you can reasonably talk about what kind of "engineering effectiveness" work you need, and its priority relative to the other tasks on your list. How much time you spend on it then depends on where the task falls on https://xkcd.com/1205/ as well as how much time you can spare from your current workloads.


Once upon a time, there were multiple repos and multiple tools for different jobs.

Then Twitter grew to thousands of engineers, many of whom were from Google. Now everything is being converted to enterprise Java in a single, monolithic repo.

Is that an improvement or just bureaucratic standardization and cargo-culting from engineers raised in the google3 monolith?


Minor side-issue: 2014 WC final Germany/Argentina; Brazil/Germany was in semi-finals.


I was constantly being reminded of Taylor, while reading the article. Is this the scientific management of software engineering? It looks a little too constraining for a rapid-changing discipline as SE.


is it just me or this article points to a huge lack of technical leadership inside twitter?

I wonder how much allowing things drag for so long cost the company? millions in salaries?


Offtopic, but why is "monorepo" pluralised as "monorepi", while "repo" is traditionally pluralised as "repos"?


I'm guessing the author is poking fun at the idea of having more than one 'monorepo' by giving it a deliberately silly plural.


The plural of "bordello" is "bordelli", so there's that. (And that's all the Italian I know.)


I wish people would learn what the hundred flowers campaign was and not use such dumb and insensitive titles for their blog posts. It's like calling it "the final solution to the twitter problem".


I wish people wouldn't take offense on behalf of other cultures they don't know much about.

Nobody much considers "let a hundrend flowers bloom" (as a phrase) equivalent to the "final solution", not even in China. This includes people who know all about Mao, the cultural revolution, the power plays and the executions that happened at the time.

It's actually a stock phrase in journalism and writing in general in lots of European countries (I know that's the case of its use in at least 4 countries/languages).

This looks more like someone in the US, several times removed from the events (by ethnicity, nation, time, etc), delivering a paternalistic lecture on foreign history.

Examples of casual use in both US and UK:

http://www.economist.com/news/britain/21581984-inner-city-sc...

http://www.washingtonpost.com/archive/politics/1986/11/03/we...

http://www.theguardian.com/politics/2015/sep/16/national-ant...


Pointing out that this phrase might carry extra meaning beyond its literal interpretation != taking offense.

But in any case, "Let a thousand flowers bloom, then rip 999 of them out" is much more evocative of those events than the casual uses you cite.


>Pointing out that this phrase might carry extra meaning beyond its literal interpretation != taking offense.

That's correct, and I wouldn't mind if it was just that.

But OP's "I wish people would stop" etc and calling it "dumb and insensitive titles" goes a few steps beyond "pointing out that this phrase might carry extra meaning".


Good point. I do think the "then rip them out" part moves the title into dumb and insensitive territory, but we may just have to agree to disagree about that.


"and not use such dumb and insensitive titles for their blog posts."

It's reasonable to infer offense from that statement.


Pointing out possible offense is almost as disruptive to conversation as announcing actual offense, so it should usually be reserved for very clear-cut cases.


I notice use of the phrase since I'm aware of the origins and the the fact that oppression under Mao has affected people close to me.

I don't take offence, but may suffer injuries related to involuntary eye rolling. It's an overused phrase and I'll assume the author is a hack and stop reading. Sort of like anything with "cyber" in the title published after 1993.


I don't think its as stock as "a modest proposal", and yet, there will be people who have no idea about the origins of the phrase.

Likewise, I think it's disingenuous for readers to begin their indignation at every corner they find someone who is ignorant of the baggage a phrase or word might have with other people. I sometimes wonder if these kinds of oversensitive people don't go about putting a list together of all the inventions and things furthered by disgraced people because partaking in such things would give such detestable things their imprimatur.

So and so was a philanderer, this other one was an opportunist, that one was a bigot, this one was sexist, that one a hedonist, an atheist, a Christian, short, stinky, tall, too cultured, not my type, too much my type...


Yikes. I'm a little ashamed to admit I had to look this up: https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign

Hopefully this blog post's author was unaware of the hundred flowers campaign. In any case it's very helpful of you to point this out so this meme doesn't get passed along by those of us ignorant of this bit of history.


I've never heard the "hundred flowers bloom" phrase before. I've never heard of the history to which it applies. I don't know major political events of the 1950s for most places in the world. The post WW2 era was a tumultuous time.

Now you aren't wrong. But I also don't think it's fair to call dumb and insensitive. It's not dumb because it's a perfectly reasonable and useful analogy. And I don't think it's fair to call it insensitive because the author almost certainly didn't know the 60 year old foreign history from which it is derived. That's not someone being callous or insensitive.

It's just good old fashioned ignorance. But ignorance that's kind of ok and fine. Like, it's just totally ok for someone not to know the origin of the phrase.


Did you go to a tech school or something instead of a real college / high school?


lol.


I wish people would learn that words and phrases can acquire meanings of their own, independent of original meanings, origins and associations.

"Let a thousand flowers bloom" has an entirely clear and positive meaning for the vast majority of English speakers.


Unless you know about "let a hundred flowers bloom", then it immediately acquires a meaning at best sinister. Ignorance should not be an excuse for callousness.


Actually it is an excuse. Callousness is about intent. Someone isn't being callous if they have no idea.


Shouldn't be surprised to see so many people tripping over each other to be offended on this site. I'd think that the same diverse group would also be cognizant enough to know that not every person visiting this site is going to know the history of every ethnic group in the world.

It's an blog post dude, not an insensitive advertising slogan for a new foreign market, get over yourself. Assume he didn't know any better and move on.


> Ignorance should not be an excuse for callousness.

Oh yes, someone is callous unless she learns the sensitivities of every person and group in the world, and then avoids them all whenever posting anything on the web.

Or, you know, we could just be more forgiving and understanding of each other.


Perhaps, but then "rip 999 of them out" doesn't just add a more sinister tinge, but also confirms the allusion for those who are aware of it.

If memory of my history class serves, the quote of letting flowers bloom comes from Mao, who told people he wanted them to open up and share their political differences (ostensibly so that some kind of political dialog could follow), but with the ulterior motive of merely outing his political enemies in order to do terrible things to them.

It's a shame to be discussing this instead of the article, though, since I am a huge Peter Seibel fan!


The fact that the comment to which you replied is the top comment of this discussion and there is so much argument below it about the phrasing of the title while the actual content of the article gets kind of ignored should be a hint that it is a poor title. It clearly demonstrates what a derail it is.



Do you really believe Mao was the first to ever use something equivalent to the phrase 'Let a hundred flowers bloom' in any language? That thought, and that expression, has many sources older than him.


As, no doubt, does "final solution." But that's really not something you'd want to use casually.


Endlösung is a word I would not casually use. 'Final solution' is not a phrase I have ever thought twice about.


I would suggest you think twice about it now, then, because if you use "final solution" then you're going to cause a significant portion of your audience to immediately think of the Holocaust. If your intent is to communicate with people rather than simply be technically correct, it's not a good phrase to use unless you actually intend that association.


Well, it's all fine and well that I learn that here now, but many other non-native English speakers will not realize/understand/feel the same thing and will never learn about it. Complaining that they are insensitive isn't going to change that and is, in fact, culturally insensitive in itself.

It doesn't really make sense to me that a regular phrase like 'final solution', which has been used as a rather awkward translation of 'endlösung', would have gotten the same connotation as the word itself. I must have said "so we end up with a final solution of ..." countless times and I've never noticed a thing.


I'm just trying to help, here. If you use that phrase you are quite likely to make people in your audience think of the Holocaust, and if that is not your intent then you probably don't want to use that phrase. If you want to be upset about this, feel free, but it won't change the facts in any way.

Language is basically a crappy form of telepathy. You're essentially implanting thoughts in other people's minds. It's crappy because it's really imprecise and error-prone and the minds you're attempting to influence might end up thinking things other than what you're trying to make them think, often because the specific words or phrasing you used brought up associations existing within the target mind.

Sometimes it's impossible to avoid because the association is a personal thing. But sometimes it's widespread. In that case, it's really useful to be aware of them.

I've certainly made my share of bad phrasing in languages I don't speak natively. No reasonable person is going to be offended if you did it because of ignorance, but once pointed out, it's ridiculous to get defensive. Make a note of it and use this knowledge to improve.


It's also subject to many fallacies, not the least of which is the Inductive Fallacy.

I wouldn't use it as part of any argument as it will likely lead you to miss important details and cannot lead you to valid conclusions.


communication is about your audience as much as about the message. using phrases that have additional meaning without knowing or ignoring it is either ignorant or counter-productive or both.


A lot of non-native English speakers are ignorant of many additional meanings and if you think less of them because of it, you are the one who is wrong.


And yet those additional meanings are really there, and aren't going to go away. So the correct response is to educate those non-native English speakers, as you have opportunity, when they stumble into unintended meanings. Don't think less of them, but also help them to not make that particular mistake again.


Plus, Americans don't learn Chinese history outside of fireworks and dragons and ancestor worship and the silk road.

Being upset about using this phrase is like an army of tumblr SJWs being annoyed when someone says gypped. Not everybody knows the complete cultural and racial history of every innocent common phrase used today, even if they come from questionable historical pasts.


They said "I wish people would learn...." To say that people don't learn about it is not a sensible counter to that!


I don't think seiji is trying to counter your post; "Plus" is used to add to something you agree with.


They're certainly not countering my post, since there isn't any of mine to reply to.

Here's what I see: jordigh says he wishes people would learn about the hundred flowers campaign so they won't make such bad allusions by accident. coldtea counters this saying that it's not actually bad. seiji agrees with coldtea and attempts to offer further support by saying that people don't learn about this stuff anyway.

Not so?


Then learn when new knowledge is offered to you, and apologize if you offend (not "I'm sorry that you're offended", all too often the clarion call of people who think "SJW" is a pejorative), and be better as a person afterwards.

Ain't that hard.


It's funny, but I feel a strong aversion at apologizing for offending someone when I didn't mean to. (I'm not being sarcastic).

Having thought about it, it doesn't really make sense, since I'd apologize for hitting someone even if I didn't mean to, and even if I couldn't be reasonably expected to hit them.


I think it's like this. You stumble, smack into someone, and you apologize. Your apology is for hitting them, not for meaning to hit them. In fact, you did not mean to hit them, you just stumbled.

But when the issue is offense, you don't want to apologize, because you think that they think that you meant to cause offense, and apologizing will reinforce in their mind the idea that you really did mean to do it, or at least in your mind it will reinforce that idea in their mind. And you don't want to reinforce that idea in their mind, partly because it makes you look bad, and partly because the idea is simply wrong. So you don't want to apologize.

(Note well: "you" here means "me, too, given the right circumstances".)

Worse: That may really be going on in the other person's mind. They may really think (or at least suspect) that you meant to cause offense, and therefore are determined that you must apologize as penance or proof of repentance or some such. "The good of society demands that I force this person to apologize!"

So maybe the best answer is to say something like: "I did not intend or wish to give offense. Apparently I did; I apologize for doing so." Because the fact is, the offense was real. Apologizing for it is just good manners. But apologize without agreeing that you had intent.

This presumes that you didn't have intent. If you intended to offend, that's a whole different situation. It's worthy of an apology, not just for the offense, but for the intent.


A lot of people do, myself totally included! I'm a prideful person. I try not to be, but I mean...I am. But there are ways to offend, even inadvertently, that definitely deserve a "shit, my bad, sorry." Apologies don't hurt, even my pride, all that much. Everybody screws up.

I haven't seen any "you're an asshole" type comments in here--more "aw, c'mon, man." (I frown more at whatever instructors the author had in 20th century history a lot more than the author himself. This is something that's not that obscure, but you gotta know it exists to know about it.)


Now's your opportunity to learn, then. China isn't exactly some tiny country at the end of the world.


Just to add to that, even if there wasn't such a negative connotation I think that the repetition of the phrase throughout the text is just plain awkward\ungainly




Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: