From null to a hundred in six months (3)

Or how to achieve bi-weekly releases at SaaS scale

Part III – People

This is Part III of a series, see Other Parts in the Series

Stupidity Laws

One day many years ago, must have been around 2001, when I was still involved with my
own venture, I happened across a short pamphlet tantalizingly named The Basic Laws of Human Stupidity [1]: it was to be one of the most influential readings I ever came across (probably, second only to Richard Dawkins’ Selfish Gene book who completely shook my entire world view at the tender age of 14).

To fully understand my attitude towards my co-workers, and my approach to building fantastic teams, one must fully understand (and embrace) the Fourth Law, which, grossly oversimplified, bars a fully-reasoning individual to associate himself with stupid folks.

The other well-known, oft-cited, but no less true, maxim here in Silicon Valley is never to hire B-players, as they will invariably hire C-players; and from there to a spiraling slide of doom towards mediocrity is only a brief step [2].

Put the two together and one cannot but logically arrive at anything other than my
Basic Law of Hiring:

Only hire people who are smarter than you, at least in one area, 
and ideally many.

This works great for me both as an individual (I get to be surrounded by great guys from whom I can learn daily) and as a leader (I get to have a team who does awesome stuff, even though I could be, for all intents and purposes, the village idiot).

Which also has a corollary (let’s call it the Corollary of Zero Tolerance):

Be relentless and ruthless in removing from the team those 
individuals who prove to be mediocre, lack initiative and/or refuse
to learn from their own mistakes.

Back to Principles

Notably absent from the principles above are any references to specific technology, areas of expertise or years of tenure: I am mildly amused by those job descriptions which require “at least 7 years Java EE development experience”; well, hate to break it to you, but I feel I must: the fact that you’ve been banging out EJB’s for the past seven years, does not make you a great developer (let alone say anything about you not being a mediocre individual).

This is instead what I look for when building a great team:

  • abstract reasoning, and in particular good understanding of algorithms (both theory and practice);
  • appreciation of testing (particularly unit testing) and a great grasp of the different types of testing (unit, functional and integration) – and their relationship to Continuous Integration (CI);
  • desire for learning new technologies, facing new challenges, testing untested waters;
  • proactive attitude (or, by negation, its ugly sister: waiting to be told what to do).

Coding Question

At SnapLogic I earned the nickname of “Code Nazi” (something that I’m only marginally less proud than of being called a “grumpy old bastard” at Google when I ranted about the invasion of privacy involved in Google+’s now-defunct precursor, Google Buzz [3]) for a good reason: as I explained in Part 1 “Code is King“, and one must first and foremost be a guiding light for the team and enforce consistently certain basic principles.

As part of my intransigence, I ask of every candidate for software development roles, regardless of seniority, past experience or credentials, to submit in writing a response to a coding question as part of the interview process.

I firmly believe that anyone unable (or unwilling) to prove that they can be a great software developer in the only unequivocal way that is possible (by writing great code, even to a seemingly trivial question) prove to me that they will not fit well within my team.

This has consistently served me well: folks who have responded enthusiastically to the challenge, seeing that as an opportunity to shine, and have joined the team, have proven to be fantastic additions to our collective brainpower and are much enjoying being part of a really challenging bunch.

Some of those who have sort of “snuck past”, when, against my better judgement, I’ve given in to bouts of self-doubt – well, they’ve really struggled, and are almost invariably eventually parted company [4].

If you feel I’m a demented misanthrope and an impossibly demanding obsessive individual, first, thanks!, and second, please do a Google search for fizzbuzz (look for Mr Atwood’ entry).

Distributed Systems

Writing distributed systems is hard.

It is even harder when it needs to be done at speed, with changing requirements and without the benefit of extensive field testing – to this, add the complexity of deploying your system to a public (or private) cloud, and its attendant configuration and maintenance issues, then it’s clear that you need a special kind of person to really thrive in this environment.

My empirical observation is that “hackers” (in the pejorative meaning of the terms, as folks who solve the problem immediately in front of their nose, without any consideration for even the simplest generalization to a broader class of problems) fare particularly badly in this environment, while people with a capacity for abstract reasoning, and an appreciation of the imperative to build an extensible and maintainable code base.

This is also why I put much emphasis in the “algorithmic ability” of candidates: problem-solving, tree and graph traversal, optimization and sorting problems, these are classes of problems that we face daily and require some intellectual rigor.

More about what to look for when building distributed systems in Part 4

Testing, Testing, Testing

I still have bouts of uncontrollable laughter (or depression, depending on the mood of the moment) when I recall this guy (reportedly a senior ex-Cisco QA superstar) who, when pressed to come up with a plan to reduce our (failing) QA cycle from several weeks to days (and eventually hours) first refused to even contemplate the possibility it could be done, but then came up with a grand plan to hire 17 QA testers – this for a development team of less than 5 at the time!

Of course his suggestion was to offshore this to India: as if, by adding a 24-hour round-trip
communication delay, language and cultural barrier and team separation to the problem, it would somehow magically disappear – because, you know, if you really want that baby, just offshore it to nine Indian women, and it’ll come to you in one month [5].

The astoundingly talented (Indian) woman who now leads the QA team is instead working feverishly to increase the level of (surprise!) automation (especially around our integration) tests, to the point where we hope (in the short term, at least) to reduce the release/QA cycle to a couple of days and possibly less, with a grand plan of reducing it to hours in a few months.

More in general, I look for folks that have an immediate appreciation for the absolute necessity of automated testing (at all levels: unit, functional, integration) in order to manage the incredible complexity of distributed, loosely coupled systems and maintain speed of development: in particular, unit testing has an almost symbiotic relationship with code quality – code that is not well unit tested (or that is not easily testable) has almost certainly bugs, is probably not well written and is virtually impossible to refactor.

More on testing in Part 4


Here’s my recipe to build a great team:

  • hire only great people, smarter than you;
  • get rid of low performers: whatever their contribution (if any) may be, you’re better off without;
  • look for people with strong critical skills and abstract reasoning abilities;
  • always ask for a coding question (in writing, or in person on a whiteboard) and be wary of weak responses;
  • weed out people who are not obsessive about testing and do not understand the value of automation;

and, oh, yeah: be a nice guy – no one wants to work for an asshole.

Other Parts in the Series


[1] In a mesmerizing twist of fate, Prof. Cipolla’s estate connived with Il Mulino (an Italian
once-reputable publishing house, now clearly running out of cash and ideas) to fantastically prove the first and fourth law in a single blow: tragically ignoring the Internet’s ability to route around stupidity, after almost 40 years of the late Prof. Cipolla sharing freely his thoughts about human stupidity, these fools are trying to make a few bucks by turning a brief essay into a “book” (it’s on sale on Amazon for the outrageous price of $8) and are using ‘take down notices’ to have the contents removed from where it used to be freely hosted.

We cannot but look in mild amusement, gently shake our heads, and move on with our lives.

[2] This happens with a regularity so impressive that it’s hard not to take it for granted as a fundamental law of physics: I’ve seen it happening many times, in almost every organization I happened to be at the time; find an incompetent idiot, and I’ll show you a stupid hiring manager.
The big difference between great team leads and mediocre ones is not so much how well they get the good guys to deliver (they’re great guys, they’d deliver even if working for a drunk squirrel), but how do they act when faced with poor performers.
[3] The product may be dead, but the attitude towards individuals’ privacy most definitely hasn’t: this is only the latest example of Google having completely lost its “don’t do evil” ways.
[4] It wasn’t funny: at some point, at SnapLogic, my then-VP (an awesome guy, whom I thoroughly enjoyed working with) desperate to fix an apparent failing recruiting strategy, in a very rare moment of micro-managing folly, succeeded in putting me off from asking a coding question as a required criteria to get to onsites: after much wasted time in depressing interviews with folks who really had no business in being “Senior Java Developer” (despite what their current title and their resume stated) and one or two hiring misadventures, we quickly reinstated it – and that was that.

Note to the wary: if you can’t find good people, lowering the bar is usually A Very Bad Idea.

[5] For the record: we do offshore to India, where we’ve found a very talented team and we’re doing some great stuff together; this is not about offshoring (about which I have my own misgivings, but for entirely different reasons) it’s about the stupidity of the idea that adding more manual testing one can actually reduce the QA cycle.



4 thoughts on “From null to a hundred in six months (3)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s