Saturday, March 22, 2008

"We Can Live With the Way You Handled That"

Tilton's Lemma had come under attack and the title of this entry was the happy result. You forgot Tilton's Lemma? Here ya go:
  • Refinements to requirements cannot vary fundamentally from the original.
The point is that careful design is worth the effort so that software can be successfully rolled out even though users naturally produce RFE after RFE during user acceptance testing. No matter what they come up with the software will adapt to it with a minimum of refurbishing.

So what was the attack? We must begin with the launch meeting of the relevant project, where my conduct was such I rarely again got asked to meetings, but as a tease I will share that this was one of the most profound God is speaking to you experiences I have had in software development, one of those moments where one senses forces unknown at play.

I was being given a nice new standalone task perfect for a cowboy contractor such as myself who was good with code but not with teams. My manager Frank would be perfect for such a loose cannon, one who cared not about how loud I snored when I catnapped or at what hour I showed up or how badly my lunch reeked as I ate it over the keyboard but only if the code worked so he could sleep at night.

Just one problem. I walk into the launch meeting and sit down having no idea what is happening. After thirty minutes of sitting quietly at one corner of the table having nothing to do with any of the discussion I have deduced that a foreign branch of the bank has accounting software that will be shut down and covered by the software at our central location and that I will be writing the interface to take the feed from an off-the-shelf humongoid accounting package and feed it into the central system in such a way that it replicates their existing system.

I have grown increasingly intrigued at this prospect as the meeting drags on because our group has more than a few people assigned to supporting the massive off-the-shelf system (shall we call it Isis?) and it seemed worthy of months of apprenticeship. On top of that I certainly had no knowledge of the foreign branch's system. Best of all I knew less than nothing about accounting. Okay, no, best of all was that no one was talking to me at the meeting so I developed this crazy idea that they were just going to turn to me at the end and say, Need anything else, Ken?

Then Tony the Senior AVP turned to me.

"Need anything else, Ken?"

Left to my own devices for the first 99% of the meeting I had had time not only to deduce its ending but also prepare an answer. My fans from comp.lang.lisp can tell you His Kennyness likes nothing better than a good analogy and it had occurred to Himself that the developing situation resonated strongly with an image that formed spontaneously in his mind of a couple with a dog on a leash standing on some corner in the middle of Manhattan deciding how best to get to the Botanical Gardens in Brooklyn and after considerable discussion and perhaps even a consultation with maps street, bus, and subway settling on a means and path of transveyance and turning to the dog and saying, "Lead on, Fido!"

Hence my prepared answer, before not one but two AVPs and the users on conference call from Canada:

"Woof! Woof!", I offered, and looking back I regret only not having thought to wiggle my ass in my chair to signify enthusiastic tail wagging.

When it became clear that the import of my barking had eluded those present and had at least stunned Canada into silence I expanded on it by observing that I knew neither the sending application nor the application I was meant to replicate nor the entire subject of accounting and concluded by wondering aloud if the guy outside tending the falafel cart might not be a better fit since he at least knew how to make change.

The Canadians had heretofore been utterly thrilled at the idea of having their accounting taken over by our central gang that could not shoot straight (I'll wait while those of you who missed the irony go back for a look) and now they had White Fang as the contractor assigned to the task, confirming their worst nightmares.

They agreed to provide a thorough specification and the AVP and I ever after when meeting in the hallway would bark at each other in greeting and at his going away dinner -- well, after the second round of shots it was pretty much non-stop howling, thank god for private rooms, we should have had one. I digress.

Remember: this is about users not being able to change requirements discontinuously. So now we know how I ended up with the most beautiful requirements from which I have ever worked. Requirements? Bah, this was pseudo code. Just brilliant. They identified a dozen kind of transactions/situations/circumstances I could expect from Isis and they specified exactly how each should be handled.

And this "exactly" was not just exact. There was this tremendous regularity in how they specified the handing of each case. Each was expressed cleanly as so many debit this, credit that, debit this subtransactions. These subtransactions just begged to be viewed as microcode for the larger instructions specifying how each case was to be handled.

So that is the code I wrote, after effusively praising Team Canada to themselves and anyone else I could find. The high-level code read Isis transactions, identified the high-level case, and invoked its handler, each handler dispatching one or more microcode subtask handlers. We are talking easy work and soon we sail into user acceptance done thoroughly for a couple of months because I mentioned they did not trust us. I also know it was a long time because by the time the call came through with the title of this entry I had forgotten about the entire issue to which they referred, which was this:

After a week my bugs had been rooted out and IRs were being handled by letting them know how they had screwed up the test. Frank was happy, I was snoring, life was good, we had time to buzz about this new write once run anywhere language called Java that would let browsers run applets.

Then well into the test when the IRs had slowed to a drip I got a call from Canada. I had blown a transaction. Lemme see it... OK, sorry, that is what the spec says. No, these have to be handled this way, the debit goes over here and credit goes over there when X is Y.

Now as usually happens in these deals I still did not know anything about accounting but some of the words had taken on glimmers of meaning and I asked, But isn't this the same as whatever? And then I threw around some terms and said that from my poor understanding what they were suggesting seemed wrong. Nope, that is how we do it. OK. Bye. Bye.

And now comes the good part. I invoked Tilton's Lemma -- not because Kenny could not see the legitimacy of their accounting, though that encouraged me, but because of something else I found to be strangely compelling: my code would not do what they wanted.

I tried to see how I could patch in the exception and there was no way. I would have needed action at a distance and Einstein had already spoken on that. What was needed was for the microcode to behave differently based on information (X and Y) available only at the top level in which the higher level transactions were being identified and their handlers invoked. The only way to give Canada what they wanted was either to add a parameter to everything in the call chain (which I would have called woof-woof, I think) or even more elegantly to set a global variable at the higher level that this one microcode handler could watch out for. Ewwww.

Now please do not think I am posing as a person of integrity or principle or anything because I am the most miserably pragmatic soul you can hope to meet, but I got back on the phone to Canada.

"Marcia, Kenny. My code does not want to do what you want, eh? Your spec is pretty clear on all these things and just from the sound of it it seems the way I am handling it now is right anyway. I can kludge things up to get what you want, but I will hate myself in the morning."

Marcia said she would talk to Bernie, her VP, and get back to me. She did not do so for so long that I forgot about it and to her credit she did not even have to make the call but she did.

"Kenny, Marcia. I spoke to Bernie. The way you are handling that transaction will be OK. We can live with the way you are doing it."

Tilton's Lemma lived on. Somehow clean design (starting I freely acknowledge with a great spec) had flushed out a faulty business practice. The users had indeed offered an RFE discontinuous with all their other stated requirements, but it turned out to be such an egregious error that they agreed to fix it. (To their credit again!)

Why does this war story give me goose bumps? Yes, I need to get out more. And OK, maybe it is not such a big deal. The careful architecture forced on us by having to program these godawful machines makes it easy for programmers to see inconsistencies the business experts miss.

I just get a kick out of it -- a sense of God or Plato or the Tao throwing their weight around as long as we take design seriously and hop on the phone over such things.


Sohail Somani said...

THIS HAPPENED TO ME TOO! Except I was in Canada talking to Americans!

Reginald Braithwaite said...


Absolutely terrific, thank you.

Apoch said...

I wish we got that kind of client. In the gaming world, everything you do will be bitched about by someone - even if they asked for it explicitly.

Tom Woz said...

Great post.

Concerning the side talk of Java applets and "write once/run anywhere"...

Did you have any profound insights during Java's infancy (and all of it's hype) which later proved to be highly accurate?

Just wondering. :)

Justin said...

Great story. You could really use an editor, though.

Kenny Tilton said...

I wish we got that kind of client.

Yep, real pros.

...will be bitched about by someone - even if they asked for it explicitly.

When that happens I always think of the boss Seigfried in the James Herriot books and feel better.

Did you have any profound insights during Java's infancy (and all of it's hype) which later proved to be highly accurate?

Nope, just that (in not so many words) Web2.0/RIA would be huge. Who knew it would be Lisp (JavaScript) instead of Java?

And it was so early for Java there was not much hype besides the run anywhere bit. By the time Java got big I was well into Lisp and could not take anything else seriously.

You need an editor

Examples? Not that I did not see a couple of things last time through, just wondering what you saw.

Reginald Braithwaite said...

You need an editor

Great, I can get into a proxy flamewar! I can pretend to defend Kenny while secretly I am motivated by shame over my own gasbaggery!!

Emperor Joseph II: My dear young man, don't take it too hard. Your work is ingenious. It's quality work. And there are simply too many notes, that's all. Just cut a few and it will be perfect.

Mozart: Which few did you have in mind, Majesty?


Tim Stewart said...

"The only way to give Canada what they wanted was either to add a parameter to everything in the call chain".

Couldn't a dynamic variable have solved this or am I oversimplifying things?

Kenny Tilton said...

Couldn't a dynamic variable have solved this?

Not in VAX Basic. :) But sure, that would be a global set then cleared around a call to the handler tree.

When I said my code could not handle it, I meant without cruel and unusual punishment (such as a global created just to get the microcode to behave differently in just these particular circumstances).

And of course I offered to do it, but being challenged a second time the user agreed to double-check with her boss.

Anonymous said...

Hey Kenny,

Great post. I enjoy reading your blog.

Martin L.

FCC said...

Could you write more about what made the specifications so good? That seems key to your story.

Kenny Tilton said...

Could you write more about what made the specifications so good?

Just what I said, that it was more like pseudo-code. They did not leave anything to my imagination, perhaps because of the barking. Or maybe it was an easy problem to specify and I am giving them too much credit. But there was no fuzzy natural language to sort through: they specified the conditions on the input data to identify each of the kinds of transactions, and exactly how to handle each, again by saying which buckets got which amounts.

My only contribution is what I would expect of any good programmer: hey, those look like subroutines (what became the so-called microcode). That is what made the code hard to bend out of shape -- had they been redundantly coded in each handler I could have just inserted their desired tweak at the top-level where the data they saw as distinguishing was still visible. Down in the microcode... uh-oh, no can do.

Ron said...

dude, good story. this is what makes hope less than psychotic. every now and then things go the way they should.

stay away from editors... they would eddy your flow...

Paul Nakata said...

This is probably not the best place to place the comment, but I don't want to mailspam you.

Have you taken a look at Clojure at all? I'd really be interested in seeing your honest opinion on it. I think you did a pretty great job taking a look at Arc.

Kenny Tilton said...

Clojure? No, sorry, I have not looked at it. The reactive agents sound interesting but the Lisp-1 and Schemish agonizing over nil vs. false do not. Not sure I am crazy about running on the JVM either. Meanwhile CL floats my boat pretty well so I may be the wrong one to ask -- I just do not see a need for new languages unless someone comes up with something so radically new CL cannot handle it. But my Cells hack is pretty far out and CL handles it fine, so it won't be easy stumping CL.

Paul Nakata said...

Thanks for the quick answer. I should have known that I could have applied the Cells / Theory Y Algebra litmus test to any lispy language to have already known the answer :D

I suppose the best thing about CL is that it gives you the tools to pretty easily evaluate other lisps for advantages and disadvantages.

Again, thanks!

Anonymous said...

Moar blogging plz. The public clamors for tales from the trenches.

Kenny Tilton said...

OK, OK, I'll try to dash off a micro-war story tonight. Nothing profound (no TLOP) but the denouement has a bit of the spooky message-from-god thing going for it. Look for either "AA, BB, CC, and DD" or "Cool! I gave up on finding that."