August 2017

There’s an interesting discussion here on waterfall project management– answering the question “Why did it fall out of style?”

113 Mill

Good thing I didn’t have to find a picture of an agile

A couple people make the point that “waterfall” was invented as a straw man by Winston Royce. He claims that the main problem with the supposed method was that testing only occurred at the end of the process, so that redesign was prohibitive.  This is absurd; it’s on the level of saying that Agile projects never think in advance what they’re going to deliver.

The first answer on that page is basically party line nonsense. Dude complains that “5 years, tens of millions of dollars, hundreds of people, not a single user, the code never ran once”, and somehow we don’t do that anymore.

The best response to this is to go re-read Frederick Brooks’s The Mythical Man-Month, published back in 1975, a set of ruminations on why big projects go wrong which everyone has read and no one ever takes to heart. Note that Brooks’s book is an autopsy of several huge 1960s IBM projects which finished, but didn’t go well. Nowhere does he say the problem was that testing was put off till the end; Brooks advocated a thoroughness of testing and scaffolding that few developers could match.

The basic idea of waterfall programming is that you write a requirements document, then a full system specification, then an internal architecture, then code. I don’t think anyone ever has believed that these processes are hermetically sealed.

And when it’s done well, it works! We used to actually do all this when I worked at SPSS. There was a design department responsible for the specifications. They worked out all the functionality, every statistical function, every dialog box, and wrote it all down. Because they were a dedicated department, they could be experts in the problem domain and in user interface, and they could work pretty fast, and get going before the developers were fully staffed. The design document wasn’t just for the coders; it was for sales, management, documentation, and QA. All these people knew what was in the system, and could get going on their jobs, before the program was written. There were big meetings to review the design document, so everyone had a chance to offer input (and no one could say they were surprised at the design).  More on the SPSS process here.

So, waterfall works just fine… if you do it. The problems people perceived with the process are not due to something wrong with the model of design/code/test.  They’re generally due to not following the waterfall method, i.e.

  • not actually doing an external design
  • not actually doing an internal design
  • not even knowing the features when coding starts
  • not actually planning for testing
  • coders generally being pretty bad UI designers (see: Alan Cooper)

Now, maybe most workplaces and devs are hopeless, and can’t do the waterfall process, so they should do something else.  But it’s not really the case that the process “didn’t work”.  It works if you let it work.

As a cranky middle-aged old guy, I’d also answer the original question (“why did waterfall go out of style”) with “Fashion.”  Every ten years, someone comes around with a Totally New Programming Style. In the ’90s it was object-oriented programming; in the ’00s it was Agile. If you’re in your 20s or 30s, you can get very excited over the latest revolution and you’re eager to make it work.  If you’re in your 40s or 50s, you’ve seen several of these revolutions, and they never live up to their original hype.

This isn’t to say that there aren’t advances! I still think OOP was a pretty good advance; it made people think about design and re-use a lot more; plus, if done well, it had the huge advantage that any given routine was small, and thus more likely to be correct. (On the other hand, done badly, it produces huge swaths of nearly-useless layers and increases complexity.)

I haven’t actually worked on an Agile project (I write books now), so I can’t say if it works out well in practice.  From what I’ve heard, it has at least two fantastic ideas:

  • Keep the code buildable.  As developers are constantly breaking things, this seems like a great discipline to me.
  • Keep close (daily) contact with everyone. I’ve had too much experience with developers who were assigned a task and it was only discovered 2 or 6 months later that they couldn’t get it done, so the early warning also sounds great.

But the insistence on short sprints of work sparks my skepticism. There really are things that take a year to program. Yes, you can and should break them down into pieces; but those pieces will probably not be something deliverable.

I’ll give an example: one company I worked for did insurance rating software. This involved an incredible number of tiny variations. You’d have general rules for a line of insurance; then state variations; then variations over time; then variations between companies.  Our original method was to write a basic set of code, then write snippets of code that changed for each state/version/company. Writing all this took a lot of time.

Eventually I decided that what we needed was a specialized rating language– basically something that business analysts could handle directly, rather than coders.  It worked very well; 15 years after I left, the company was still using a version of my system.

The point, though: writing the language interpreter, and the proof of concept rules for one line of insurance, took time. Half a year, at least.  There was no way to divide it into two-week sprints with customer deliverables, and daily meetings would have just got in my way.

I can understand that doing things this way is a risk– you have to trust the programmer. On the other hand, I’d also point out that I took that project on after about six years at the company, and it was my third architecture for the system. (Brooks rightly points out how second projects go astray, as programmers do everything they ever dreamed of doing, and the project gets out of hand.)



Maximum City sounds like a good name for a non-Big-Two superhero comic, but it’s a book about Bombay by Suketu Mehta. (The city is officially Mumbai, but Mehta prefers the old name.)

It’s amazing. I reviewed a few other books in the Gawking at Modern India genre,and this is by far the best.

Mostly, it’s because Mehta dives down, deep down, into the underbelly of the city. A lot of the book is about the gangwar— one word in Indian English, pronounced like gen-gwar. He finds the hit men and hangs out with them. He’s full of factoids about them. They’re mostly slumdwellers (but then most Bombayites are; we’ll get back to that). It’s a good job for them; most of the time they hang around watching TV or talking, between jobs; they have plenty of money to spend on girls.  They love Bollywood movies, especially the ones about gangsters. They’re often quite religious… when death is on the line, you’d like some supernatural aid. They seem to like talking to Mehta– they like the idea that their story is being told accurately.

Mehta talks to one of the highest bosses, who runs the gang from Karachi. I wish superhero comics writers would read this, to understand how a powerful villain really talks.  The boss is gracious, solicitous.  He wants to appear normal and rational.  He offers a free contract for Mehta: he can have some work done.  That’s the slang for offing someone.  Mehta muses on what it’s like to be able to call up someone to have someone else killed.  He says it’s calming– you feel a little more magnanimous about Bombay’s many irritations.

He spends almost as much time with one of Bombay’s top cops.  Though honest and effective, he’s also about as brutal as the gangsters. He doesn’t much bother with detective work, though he uses plenty of informants; he picks up a gangster and tortures them. If the police have had enough of a gangster, it’s far too unreliable to try to prosecute him– a bribe will quickly free him.  They just shoot him.

The metropolitan area has about 20 million residents.  9 million of them live in the slums. This is largely because of a well-meaning, disastrous bit of legislation: the Rent Act of 1947. This froze rents at their 1940 levels, and prevented evictions of tenants. Naturally, no one who has an apartment ever moves out; when they die, their heirs inherit the place with its frozen rent. The law was intended to be temporary, but the millions of tenants far outnumber the landlords, so all parties support the bill and it’s been extended till the present.  (Mehta notes that India is the opposite of the US, in that poor people vote, and the rich do not.)

Landlords can’t make money at 1940 rents, so they refuse to maintain the buildings. Services and pipes and wires fail; the tenants repair them haphazardly. There’s no way to make new buildings for profit, so none are built. Instead, people build their own houses in the slums.

Another long chapter is on bar girls.  In the West these would be strippers, but in fact the dancers don’t take off their clothes. Men shower them with money anyway.  They press to get the girl’s number; she is expert at slowing them down, only slowly relenting to give a man her phone number.  She will accept calls, and then only accept meetings if he’s generous with gifts.  It’s not respectable, but there’s far more money in it than in the respectable jobs.

Mehta frequently wonders how Bombay works at all.  Everything is slow, corrupt, falling apart.  Gangsters extort money from businesses; nationalist politicians keep the population at the edge of riot. And yet people keep moving into the city, eager to get in on the action.  Bombay produces 25% of India’s industrial production and 40% of its foreign exchange.

While living in Bombay, Mehta managed to write a Bollywood film, Mission Kashmir. This was done in story sessions with the director and other writers. Though there was a script, you don’t get actors to appear in your film by sending them a script.  You go to their house and basically act out the story for them.  Oh, and banks won’t finance movies, which are highly risky ventures; the underworld does.

There’s a chapter, full of ironies, on the Jains. They are big in the diamond industry, and thus many are very well off.  But many of them choose to give away all their wealth and live as poor beggars; Mehta follows one family through this process.  The family not only gives away everything it has and wanders the roads asking for food; it splits up, as men and women monks can’t travel together.  And yet the process of giving away wealth is done as ostentatiously as possible, at a massive town-wide gathering.

More factoids for you: India is not over-dense with people, compared to Europe. The population density is greater in Belgium than in India. On the other hand, parts of central Bombay are denser than anywhere else on Earth: 1 million people per square mile.

On the lighter side: to Indianize your glass of Coca-Cola, you add spices: lemon, rock salt, pepper, and cumin.

The crime and corruption and density Mehta talks about, by the way, are not exotic features of India; American cities were exactly the same a hundred years ago.

A lot of the success of the book comes from its choice of subjects, but also from the depth and empathy that Mehta brings to them.  He obviously spent a lot of time with people, and the result is a set of portraits that feel like far more than interviews.  (And probably only a born Bombayite could have written this book, simply because he has to know English, Hindi, Gujarati, and Marathi to understand what’s going on around him.)


Last night I had an amazing game of Heroes of the Storm.  Like most good Moba stories, it’s the story of a comeback. (When you roll the other team, it may be a well-played game, but it’s not a story.) Here’s the situation 11 minutes in; we are the red team.


It was grim. They were two levels ahead, and were fighting at our core. (You lose the game when your core is destroyed.)

This is the Dragonshire map. The clever bit of HOTS, the thing that makes games last 15-20 minutes rather than League’s 40-60 minutes, is that each map has an accelerant, something that you can fight over that gives you enormous power. Here it’s a statue of a Dragon Knight which will come to life and fight for you if you can control two temples at opposite sides of the map. In the picture, the enemy Knight is Shodredux.

I’m playing Chromie, currently my favorite character.  She’s tiny but very powerful. Her Q is a sand blast that only hits heroes– thus, useless against forts and minions; however, that also means the minions don’t block the shot.  Her W hits a small area; her E sets up a trap, and her R is a fairly big sandstorm that drastically slows everyone in it.  Once she’s leveled up, her Q is devastating, and if you can catch people in her sandstorm, you can spam them with her other powers. She’s fragile and has no escape methods, though.

We killed the Knight, and the enemies behind him (some were captured in one of my sandstorms, just visible at lower left in the picture).  We fought off another attack on bottom lane, which allowed us to catch up in levels. The playback is a little embarrassing at this point, as it looks like I’m just wandering around. I ended up returning to base to restore mana.

Most of the action was up top, as everybody fought for the top temple. Well, somebody’s got to try for the bottom temple, so I went there and captured it. I was alone there… only, no, Medivh flew in!


We traded blows, and I put a sandstorm on the temple region.  That kept him there just enough to finish him off with a Q. I was low on hit points myself, so there was nothing to do but teleport home, hoping that no one else would head to that temple.

I ran out mid to the Knight statue. There had been a grand battle for this, and by the time I got there it was reduced to our Butcher holding it, close to death, against Dva and Artanis.  I snuck in and took control of the Knight.


(There was no time to negotiate this; better to use him as a meat shield. It takes about 10 seconds to take control of the Knight, and you can’t be attacked during this time. And really, mad props to SassySadist playing Butcher here: he had dispatched Artanis and got Dva running away, despite being close to death himself.)

So now we had a Knight!  I headed out mid, and the rest of my team converged beautifully on Butcher and me.  The rest of the game was a constant team fight, taking down forts and towers and the enemy team. The Dragon Knight was felled, but that just meant I was at the Core as Chromie; I set up a sandstorm and kept throwing Q’s into it.

If you know any Mobas, you can see from the mini-map how strange our victory was:


Blue still has two of its three towers.  Normally you’d take them all down. You can also see that Red has no towers on the bottom lane, due to the fight at the core described above.

Only on watching the playback for a second time, writing this post, did I understand how we did it: our little knot of heroes stayed alive, and kept picking off enemies, so we usually had a 5-4 or 4-3 advantage, and right at the 20-minute mark, a team kill which allowed us to beat up their core at our leisure.