We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on.

From the outside, I see systems that are always changing, so users can never rely on anything working the same way from one visit to the next. I see organisations with access to sensitive personal data being cavalier at best about how they handle it. I see a major blunder every few weeks that at best causes serious irritation to users and at worst risks significant loss of business and/or legal/regulatory consequences.

And I see huge brands whose egotistical staff don't realise that they are successful despite evidently not bothering with either real product management or robust testing, not because of these things, and who don't seem to notice that they are still relying almost entirely on momentum from one or two really big successes from early on to maintain their user base and revenues.

However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin.

For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting without one of our small team struggling to see either a word processor document or a spreadsheet properly, and that's just the minor cosmetic or browser incompatibility bugs that keep appearing along with all those minor UI changes. We have also experienced some much more serious problems, including corruption/data loss and seeing the entire change history for some files become inaccessible for no apparent reason. In other words, it's not just minor UI errors that crept in as the product evolved, there are evidently fundamental flaws in the underlying architecture that don't store the data robustly.

Another example: I spent a couple of days last week trying to figure out why a site that had been working fine for users until recently and had not been changed at all on our side was suddenly generating bug reports. It turned out that recent Chrome builds have broken some HTML5 features in multiple ways. There have been related bug reports in some cases, but they have been closed as the specific example given no longer seemed to be a problem. Again, the nature of the problems makes it obvious that these are not just one-liner issues but fundamental flaws, typically where Chrome is so aggressive with its cache usage and/or trigging redrawing/relayout that it just plain doesn't work. And even though the bugs had been reported, obviously the correct root cause was never identified and fixed.

Another example: I recently spent some time looking into how the Closure Tools are coming along. Have you tried the examples/demonstrations for the Closure Library recently? Many of them simply don't work in Gecko-based browsers, and that would be obvious if anyone working on the project had made even a cursory attempt to test them for five minutes.

I will finish by once again acknowledging that my experiences here may be atypical, and that the particular projects I've mentioned that I happen to be using may not reflect the wider Google culture. But on the evidence before me, I see an organisation that keeps breaking things in its rush to push new features out and that demonstrably lacks robust architectures that keep data safe, effective testing processes before pushing code into production, and proper issue resolution processes when defects do get reported.

The notion was that it was much better to be able to notice and fix bugs quickly than it was to add process delays that reduced bug frequency. That especially makes sense to me when you think about the value integral. The longer your release cycle, the longer you keep people away from improvements. It also makes sense if you think about companies as learning organizations; speed of learning is limited by the length of your feedback loops.

So I'd say that Google's mistake isn't going for a release-early, release often approach. It's that they don't pay sufficient attention after release to user feedback or user metrics. Hell, I've stopped reporting bugs to Google, and I must know 10 people who work there. Even with a good backchannel I just have no faith that anything would come of a report.

It's more likely that you just spend more time using Google software than other software. When I worked at BofA, our website was down for an entire week because of a bad push. No online banking for a week. That's pretty much the standard for the industry. I don't doubt that there are bugs in Docs or Chrome, but they are relatively obscure. That's the nature of software, not every bug is caught by a unit test and users end up seeing them and reporting them. (Oddly, we use Docs heavily at Google, and the only bug that I've noticed is the "Your zoom level is not supported one". I haven't personally hit any other issues.)

But like I said in my original post, if you know how to develop bug-free software, I'd love to hear how. Expecting low-cost web apps to be as reliable as airplane control systems is unrealistic.

Sorry, but I'm really not. In the case of Chrome, for example, I would routinely test new work on web projects in all the major browsers. I am writing this with some empirical data in front of me, because I've just checked the bug tracking systems for a couple of projects I work on to be sure: for both projects, excluding mobile browsers, Chrome has been responsible for a clear majority of all defects where the root cause was found to be a browser bug over the past two years.

I don't doubt that there are bugs in Docs or Chrome, but they are relatively obscure.

Respectfully, if they were that obscure, my colleagues and I wouldn't keep running into them on multiple projects. I'd agree that the particular symptoms of any particular bug are usually a corner case: set this option to A and that option to B and it breaks, but other combinations work OK. It's the way that several of these bugs have recurring themes that betray both an underlying architectural weakness and a lack of effective quality control that I find most unfortunate.

Using Chrome as an example again, it is clearly aggressive with caching and conservative with repainting, but sometimes that means it is simply not behaving properly at all. If I set a part of the DOM to display instead of being hidden and then send an AJAX request, I want my "please wait" message displayed while the request is running, not afterwards, or indeed not at all since it probably gets hidden again as soon as the response arrives.

That's the nature of software, not every bug is caught by a unit test and users end up seeing them and reporting them.

And as I mentioned, several of those bugs in Chrome had been reported, and subsequently closed without the root cause ever being identified and fixed.

Expecting low-cost web apps to be as reliable as airplane control systems is unrealistic.

I don't expect Google's software to be as reliable as airplane control systems, but somewhere close to as reliable as everyone else's software would be nice. I appreciate that you're having trouble believing it or reconciling it with your own experience, and I've already acknowledged that my experience might be a complete outlier, but I'm looking at several years of empirical data across multiple completely different projects and development teams and it is quite clear that the Google products I'm looking at here haven't been keeping up lately.

Link?

The issue was closed almost immediately, with some obviously hastily written comments, apparently because no-one could reproduce it in a different version of Chrome on Linux or Mac. As far as I can see, no-one even tried to reproduce the other reported failing case on Windows, and there was no attempt at all to investigate the original bug and determine how it happened and why it was no longer observable on the platforms tested.

The issue was simply marked "fixed", despite no actual fix having been identified, rather than giving it a more specific "no longer reproducible" status.

There are still serious problems with that combination of attributes today.

I agree, though FWIW I'd say it still takes more work to support Safari on iOS because of all the odd quirks. Apple might not consider them bugs. They're entitled to their opinion. ;-)

It surprises me that we've reached this point, but several of the web-based projects for business users that I work on professionally now recommend IE9 and won't officially support either Firefox or Chrome, and I agree with them. The trendy culture of frequently pushing subtle changes and half-baked new features and then hoping that if anything breaks you can fix it fairly quickly might work for places like Facebook, but it just doesn't cut it for professionals.

The issue was known and in the bug tracker several days before the release, yet they went ahead and released anyway. Consequently many sites and browser-hosted user interfaces across the world were broken.

It was then a very long time before another patch was pushed to fix the regression.

I'll leave it to you to decide whether it's worse that such a serious bug could get into the repo in the first place, that they still pushed the update to everyone even though they knew about the problem, or that they allowed sites that rely on these technologies to be broken for so long before correcting their mistake.

Broken? Depends on your perspective.

If you care first and foremost about your users' data, it sure is broken. But FB is not a hosting company.

If you care the most most about correctness of your code and getting everything right the first time (VMS style rather than UNIX `worse is better' style), then again, it will seem broken. But FB is not an academic excercise.

If, on the other hand, you felt like a stakeholder in the company and cared most about developing the business, then no, `moving fast and breaking things' is means to that end.

If, alternatively, you enjoy experimentation and pushing frontiers, then again, no, `moving fast and breaking things' is about the only modus operandi that can accomodate that.

Emphasis on product management and testing is for serving your customers -- but users aren't FB's customers. I cannot assure you, but I would bet a lot FB does indeed heavily test and manage projects related to serving ads -- which is what their customers pay for.

If I were Zuckerberg, I'd be terrified of slipping into a premature complacency or a heavyweight process that prevents change. After years of social network king-of-the-hill, Facebook seems to be set up for the generational dominance of a Microsoft or a Google. But it's not locked in yet.

Sure, current practices risk irritating users or a severe FTC beatdown. But those are obvious risks, easy to measure and mitigate. The real harm to Microsoft didn't come from annoying UI or regulatory intervention. It came from long-term shifts that they were too culturally isolated and too inflexible to understand or adapt to, much less drive.

There are certainly downsides to shipping early and often and then seeing what happens to your user metrics. But bottom-up power to ship means that a lot more good, novel ideas will see the light of day.

He just listed Facebook on NASDAQ - cant think of a single business decision that would do more to cause this result than that!

By listing on NASDAQ, Facebook at least gains some capital in return for this reporting.

Citation needed. The massive public success of these companies would argue against you, as would my personal experience working with and without traditional project management.

Huh? Do you think they started out as straight-laced businesses with traditional project managers, and then arbitrarily decided to get rid of them after they became successful? Their cultures were always like this; did you see the recent story here on how Facebook was in its 30-person days?

No, I think they each started out with one good idea that they executed well enough and I think each also had the good fortune to be in the right place at the right time. They made great successes out of their respective founders' big ideas and their first funding rounds, but that was nearly a decade ago in Facebook's case and well over that for Google.

Today, these organisations have been considered among the most desirable employers for many of the best and brightest software developers of a generation. They have more money than they know what to do with because of those original successes, and they have thousands of very smart people working for them. No-one can credibly claim that they don't have vast and talented software development teams.

And yet, neither has produced an industry-shaking development in several years, never mind creating any new markets, and their original successes continue to bring in by far the lion's share of their revenues. I don't see how you can blame that on anything other than a lack of leadership and vision from the other parts of the organisation. The people who are supposed to be guiding and nurturing and co-ordinating just don't seem to be there, and it seems they're still trying to operate like that 30-person organisation, just scaled up by an order of magnitude or three.

"Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves."

I actually like this idea. Building my own website as well as working as the senior engineer during my day job forces me to be involved in all facets of web developing. The jobs are not abstracted. You are expected to know what you're doing from the front end to the back end. If not, nothing get done in a timely manner. I like this method because the front end is tightly woven in with the business logic of the module that you're working on. In other words, you know what the code is doing inside and out.

But when I work with back-end developers who don't know MySQL inside and out, and they write a query that works fast in development but slow in production because they didn't realize the column index they specified won't work because of a string collation incompatibility between tables, and they've never even heard of this kind of problem before...

Then I wish they'd stick to writing their back-end executable code, and have a database guy write their query for them, and have a JavaScript expert take care of the front-end stuff.

It's not a question of intelligence, just a question of experience. You have to have done a lot of JavaScript coding to realize never to call parseInt(x), but always parseInt(x, 10), for example. And the number of CSS incompatibilities between browsers that you need to account for...

Unless someone has a lot of full-stack experience, it doesn't seem that great that they should be pushing out full-stack code on their own at a place like Facebook. Maybe code reviews will catch those kinds of things, though.

If there is some domain knowledge needed then I can just rope in an engineer from the appropriate team to review my diff.

It's also worth noting that on product teams engineers tend to gravitate towards the parts of development they are more comfortable with, or enjoy the most. They are not forced to work with te full stack just for the sake of it.

(I wrote this stuff before but for some reason it just didn't go through)

As far as testing wise and building things, I believe it is best to have an engineer that is able to fulfill those roles for the following reasons... 1) They understand the structures and what it is that makes an application do what it does. 2) Its a very good learn tool. If you mess up you know exactly why. You'll be able to constantly adapt yourself to become better. 3) Waiting for your "role" to be utilized within an organization may sound great because you have free time, but it causes stagnation and "Diva-ism". That is the whole "Its not my job, so I don't care about your problem" or "You need me more than I need you" attitude. 4) You can move relatively quickly since the code and logic is coming directly from you.

Yes, specialists are very important. For large organizations they are very needed. But, to get things to work right now, you need to be flexible and open-minded. Look at Facebook's UI, its not the prettiest (blue bar at the top and 3 columns), yet its an extremely popular product. Most users just want the damn thing to work.

I'm no web designer, but I understand that a clean UI is needed. I'm not a DB guy, but I can figure out most problems. Etc. etc.

I think that's what Zuck actually wants in his engineers. Guys and gals that can see a problem and fix it on their own. Guys and gals that can think of a cool idea, test it, and have it released for that 1%.

Code reviews and peer help will smooth things out. I believe it was Jamie Zawinski's interview in the book "Coders at Work" that had the same mantra when he developed Netscape. Just get the thing out fast. You'll get more eyes on it. And you can always make it better the next time around.

It seems like a lot of shops do that. Its a cut throat world and if you spend too much time trying to be super clean and precise, you'll still be on version 1 while your competitor is on version 10.

I think the right solution is the approach IDEO takes: teams of what they call T-shaped people: broad general understanding, with great depth in a particular area.

In Facebook's case in particular, I don't think they need a lot of brilliant design as much as they need solid, competent, well-integrated features. Indeed, their latest attempt at design brilliance, Timeline, has been much less well executed than their zillion engineer-led evolutionary changes.

With software, I think it's important to see the forest for the trees, so that you can make the best tree for the forest... (apologies for the tortured analogy)

That said, your comment about the front end being "tightly woven in with the business logic of the module" scares me a little. The front end is presentation and interaction. Unless most of your businesses mojo is client side, even a full stack engineer should design each layer as fully cohesive and encapsulated components. I may have misunderstood you, however.

I did write most of my ideas to "underwater"'s comment.

Basically, my position is more on the fact that if you need things done quickly, then a full stack guy is the one you need. You can fix/abstract the code later. In fact, I encourage that. But to get the ball rolling, you're gonna need some quick and useful (sometimes ugly spaghetti) code.

Its a trade off that must be made, especially you're rapidly building your product.

after boot camp, all engineers get access to live DB

I can understand on a startup or small org but an organisation of that size, there should be very tight access control.

Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else.

I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be.

I'd never put my data near FB. They are simply irresponsible.

These are rockstars.

They work for facebook.

They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';

    UPDATE users SET email = username || '@facebook.com';

You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.

I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)

One size (of engineering practices) fit all mentality is not the correct approach either.

It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc.

A list of investments or your mortgage statement is way less important.

Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.

Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.

And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.

I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence).

Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log.

I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks.

[NB Of course, none of this would be applicable to Facebook]

It just works.

In the last 5 years, we've never hooked a debugger to production instance or given access to a developer.

In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP).

That's how it's done.

There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.

Of course, I'm still filled with rage when I find a cargo-cult suite of unit tests. But it's so much easier to convince those people to write smarter tests than it is to deal with the team that has no time to write tests because they're spending all their time debugging.

My whole mantra is don't manage complexity: avoid it.

Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.

Now that I think about it, I doubt Facebook is that careful, considering how often they break things.

(Poe's Law says hi.)

Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports).

Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L.

Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.

Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume.

Madness, sure, but common madness.

Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.

PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.

http://www.quora.com/Facebook-Engineering/How-accurate-is-th...

"resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on."

That sounds exactly the same as how Github's engineers work.

It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interested in working on the project, then it's a good indicator it may not be a project worth completing for the company anyway.

I'm sure there are times when someone has to say "We need someone to do this", but I'd be curious to hear from someone who works in one of these environments how common an issue that really is.

Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data.

One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly.

Who will fix The System?

Of the five engineers who wrote it:

* Jack and Mandeep left to launch myfornaxisnatrr.com

* Wei Li has moved to a different group

* John doesn't want to touch it, he only came on board to help Jack

* Michael was an intern but has since taken a job with Google.

Uh oh! There's nobody around to voluntarily fix an existing system. That's boring, and there are no incentives for fixing bugs in obscure features because only launching successful new features gains visibility from higher-ups.

Guess we'll need some mean old managers to round up a posse.

If they care enough.

Meanwhile, The System has acquired millions of users, cost millions of dollars to develop and operate and will now abruptly cause tens of millions of customers to become incredibly frustrated. And at no point has anybody stopped and asked:

"Was this the Right System to build?"

Companies really should find a way to incentivize this. Rewrites are about as silly. "Write the next generation xyz." Translates to "No one here has any idea how xyz was written, so it's being rewritten instead of modified. We look forward to rediscovering the same problems we did the first time."

It can't be done from the top. You need to start with engineers who care about quality over the long term. It has to come from within each engineer.

When Zuckerberg talked about younger programmers being "better", he probably meant "more like me". But old farts are just young farts with expensively-acquired scar tissue.

Most of our most treasured software development lore comes from the lesson that "move fast, break things" just devolves into "fuck, everything is broken, fix it fast".

Sometimes, when software has been around for 10+ years, stuff does get too old. It happens. For real.

I know people seem to forget that here with startups having been around for less than a year being "old players" and all that, but common.

Software may have been written with a bunch of presumptions which was valid when the project was initially started, but no longer are that. Maybe it was written on top of a platform which is no longer as productive as it was when the project was initiated. Nothing kills enthusiasm and productivity like working on a platform no longer deemed modern.

And then you have technical limitations and pre-conditions. Once, servers was expensive and you wanted your solution to distribute it's load to clients so that you could save on servers. Once servers was the only machines powerful enough to complete time-critical tasks within reasonable time-frames, so you architected the software accordingly.

Now, we suddenly have HTML5 and webworkers and shit, so you probably want your solution to be a web-solution with a distributed computation model. I.e. full-circle. Full rewrite. Twice. What's next, I don't know.

Sometimes a rewrite is just right. Saying that every rewrite is based in incompetence is simplistic.

One of the advantages of the "bootcamp" approach outlined in the OP is that people feel empowered to jump from product to product, so if something you care about breaks for a small set of users and you wish to fix it, you can do that. As a few others have pointed out, it doesn't hurt to mention this at review time, and people are often publicly recognized and thanked for these "thankless" efforts.

In places where the goal is to make engineers happy and to impress other engineers, there are incentives for behaviour that are perverse from the customer's point of view.

Good software should be written on the assumption that someone else is going to be editing your code at a later date anyway. If it's highly readable, then being unable to grab the original developer is less of an issue.

This is one of the reasons for the peer code reviews during the original development.

I doubt it's really that hard to find developers to work on issues like this. I've always thought the idea that their work impacts millions of people, even though that is a tiny percentage of the total userbase like you said, was one of the big attractions of working at Facebook. If I knew of an issue impacting that many people, I'd be delighted to have solved it.

The PM/engineer divide feels uncomfortable to me, but I have limited experience with these roles, so it's likely simply for lack of knowledge. I hope someone can clarify this for me.

If an engineer has an idea, they seem to be more encouraged to just find some people to work with, build it, and then roll it out to a very small section of Facebook's users and see how they react to it. Good ideas might then be taken further, but bad ones (Or I guess, ones people just don't care about) won't be.

(I don't work there though, so I have no way of knowing if that's right or not, sorry!)

That's not an entirely small amount for Facebook since they don't have a zillion different products and features that need to be owned, like a company like Microsoft or Google.

I think that if you get buy-in from the engineers working on the project it can induce ownership and a willingness to watch it succeed and work towards it's success. The issue occurs when there isn't adequate a/b testing (or something similar), which could create the fiasco such as facebook's switching of email addresses.

Companies such as Google allow 20% time to work on projects that might not make it to production, or get killed after acquisitions. I think that policies such as those work better than code being committed on the fly.

One of the risks to consider with a "devops" oriented approach is that you may become more dependent on it than you want.

Often, applications split things into a few different categories depending on how tweakable they need to be. There's code, configuration, app administration, and data. Code shouldn't need to change often. Configuration may need to change when the environment changes. App administration (eg creating new accounts) needs to change often, and data is always changing (or at least growing).

The risk is that developers will design the system so that only developers can administer it. Configuration, the settings that may need to be tweaked by sys admins long after the original developers have left the project, may wind up in the code or sometimes lumped into the database alongside end-user options.

It's not a reason not to take this approach, just something to consider when developing internal processes and culture.

↪ How Facebook _actually_ pushes updates to the site

I came across this originally on the Etsy dev blog, rather than HN, and that particular post had some good other stuff about Flickr and Etsy:

http://codeascraft.etsy.com/2011/06/01/pushing-facebook-flic...

Years ago I did a gig at eBay, and I thought their macho attitude was a giant source of problems. Plenty of good, sane people were driven off (or driven mad) by artificially high-pressure situations. Every email about a promotion mentioned how somebody had worked all night to get something done; they were promoting more for drama than for skill. And a "this isn't hand-hold time" attitude was common among senior technical staff, which meant that people often hid their weaknesses rather than getting the help they needed.

The lesson I learned from that is that software companies that take normal circumstances with the intensity of emergencies are gradually cutting their own throats.

That's a lesson reinforced for me spending a lot of time with a family member in hospitals last year. Even when survival was on the line, the best doctors and nurses proceeded with patience and kindness, working to train staff and improve systems as they went. In an actual emergency they moved like it was an emergency. But only then. If they can be serene and thoughtful while dealing with brain tumors, I don't think there's any reason that people at Facebook have to puff themselves up with self-importance.

It also seems that an "excellence first" culture seems to provoke fears that people can't learn, or aren't allowed to make mistakes. Tone and delivery are important here - especially from leadership. My take on excellence first is you don't have to compromise environment for it. Create an environment that fosters excellence, where people aren't afraid to make mistakes but are committed to learning from them. Everything doesn't need to be treated like an emergency, but theres a level of bullshit on a technical level that shouldn't be tolerated. Again, each situation needs to be felt out individually and moderation is always key.

Personally I don't advocate the internal competition aspect of excellence either. The pitfalls of that are well documented as you have pointed out. I'd promote based on team success, and THEN individual contributions. If you can't work as a team then the whole team loses, only once the team succeeds would you start singling people out to see where the best influences were coming from.

TLDR - I think there is a middle ground between not running it like a daycare and running it like a wall street trading firm or gulag.

Facebook uses chat/pictures to sell ads and cheesy games.

Their entire value proposition comes from surfing a wave of attention. What they need are scruffy surfer dudes who keep an eye on the weather, show up at the likeliest beaches, and never stop riding the waves. Anyone with a modicum of skill who keeps paddling out to try again can make a real contribution.

What they do not need are rockstars, certified supergeniuses, arrogant hotshots, etc. You can't buy or impress your way to the big waves, you have to ride there.

If you try to keep winning by being awesome, you will wipe out. For example, Netscape, AOL, Alta Vista, Geocities, MySpace, and many others.

What it provides is trivial, how it does it is not.

The whole move fast, make mistakes mantra is a bad approach when you're a high-trafficked site like Facebook that risks jeopardising your revenue. A second of down time can be very costly. Knowingly pushing out half or completely untested features might be acceptable if you have a small user base, you're a new Internet startup or your logo says alpha or beta, but certainly not an established brand and company it's stupid.

This is all my own opinion, of course.

If you have a small user base or are just getting started, delivering the best product experience is the #1 thing you can do for success. Is iterating faster and making more mistakes and giving the early adopters (ones that champion your service the most to others) a sub-par experience worth the speed? Sometimes, yes. All the time? No.

Facebook can change your privacy settings, steal your e-mail address, recognize you automatically in photos, watch your browsing activity outside of their walled garden, and sell your data to advertisers. And yet they still have 500million+ users.

Maybe this is why Facebook seems to be chaotic for users. They change and change to whatever design they wanted without much consideration about what the user's would feel about the design. Facebook has now become the combined features of the other existing social media.

Sounds like a lot of code debt accumulating that could bite them hard down the road - it's one thing to write and manually verify bug free code, it's another for a different engineer to make sure he/she doesn't break that code inadvertently a year later when the original author has moved on to another project or company. I'm not talking about 100% test coverage; if the smoke test for a feature breaking is someone noticing while playing with the site, in the long run it strikes me as a much less efficient way to verify and fix regressions than using an automated test suite. Writing good tests is hard, but keeping a product bug free as more and more functionality accumulates without automated test suites is even harder in my experience.

Some groups put particular emphasis on this (e.g. the messages team is great about testing), and it shows in the reliability of their products. Even better, they end up building frameworks that make it easier for the rest of engineering to write tests, and drive the whole ecosystem forward.

http://www.startuplessonslearned.com/2009/07/embrace-technic...

    What's the most evil thing we can think of doing?
     V 
    Candidate <-----<------- Think of the next-most evil thing we can do
     V                                     ^   
    Can we code that? (no) ------->------->^
     V (yes)                               ^
    Is it legal?                           ^
     V (yes/no)                            ^
    Can we get away with it? (abs. not)--->^
     V (yes/maybe/prob. not)               ^
    Keep shipping!                         ^
     V                                     ^
    Did we get in trouble for it? (no)     ^
     V (yes)                        V      ^
    Claim it was a mistake!         V      ^
     V                              V      ^
    Still in trouble? (no)-> Keep feature->^ 
     V (kind of)                    ^
    Being sued for it? (no) --->--->^
     V (yes)                        ^
    Throw money at lawsuit          ^
     V                              ^
    Did we lose? (yes/no) ----->--->^

I asked this in another (semi-dead) thread and it didn't get a response, so I'll ask again as I couldn't find any information on it. Was any legal action taken against facebook in the recent e-mail hijacking fiasco? I think there should be, since e-mail has pretty much replaced snail mail in terms of business communications, which directly affects the livelihood of many people. Companies like facebook with so much power at their fingertips should not be able to get away with doing things like altering millions of people's address books for their own selfish reasons.

That page also says that a simple thanks may be acceptable, but the consensus seems to be that those comments are superfluous as well.