smuglispweeny

Fortune Cookie File Number Two

2010-08-28T18:56:00.008-04:00

Wow, what an effort! Nice followup to part one! Thx, Madhu!

%
Are you trying to win a Moron Contest*, or did I miss a joke?
-- Kenneth Tilton
%
[Quoting some "Bell Labs engineer" from the Newark Star Ledger]

"The hardest part for me was realizing I was being tolerated by all the people I had been tolerating."
-- Kenny
%
That is a metaquestion. No metaquestions allowed under Obama. Note that this is a metametaanswer, it's like minus signs, ya got two ya ain't got any. With me so far?
-- Kenneth Tilton
%
Let's decide on the shape of the table before deciding if the discussion to decide if free open software is ethical is going anywhere.
-- Ken Tilton
%
Pioneers do not wait for the rest areas on the interstate highway to
have sushi bars.
-- Kenny
%
The Mac I have now sounds like a 747 turbine after spinning up, can't
even hear The Voices let alone program a computer.
-- Kenny <490ba086$0$5643$607ed...@cv.net>
%
What part of Anna Kournikova do you not understand?
-- Kenny
%
That will be well-received, and when Anna Kournikova sends you an
email asking you to come spend a week with her you complain that she misspelled Tahiti and the airline tickets she sent were not first-class and write back asking if she knows anyone prettier you could stay with?
-- Ken Tilton
%
I think Celtk just needs Cells. This is like saying that if you want
Anna Kournikova to have your baby then you have to sleep with her.
-- Ken Tilton
%
Anna played tennis?
-- Ken Tilton
%
Muhammed ibn Musa al-Khowarizmi! He wrote the book on Algebra!
Literally. Not sure where I can get a picture, tho. Maybe I can give
Anna a beard...
-- Ken Tilton
%
I feel a naggum coming on. When you think you know what I am thinking try to stop thinking.
-- Kenny
%
[...] you should -- I feel a naggum coming on -- stop trying to impose
your prior understanding on a new experience. You are like an
American tourist landing in Kinshasha and going in search of a Burger
King. Sadly they do not have to go far.
-- Kenny
%
I feel a Naggum coming on. Before you contradict me or challenge me or quote me, please make sure you are not in fact talking about an inferior model of me rooting around somewhere in your cortex (and what a scary image that is).
-- Ken Tilton
%
New Yorkers are not offensive. The joke is a good one (The proper way to ask for directions in NYC? "Excuse me, can you tell me the way to Lincoln Center, or should I go fuck myself?") but the reality is New Yorkers are so nice they will gladly give you directions whether or not they know the way.
%
[...] the mentally ill are often the most compassionate because of what they have endured.
-- Kenny
%
Instead you are like a new yorker asked for directions and before they
can even get out the words "...to Carnegie Hall" you have responded
"Use Google Maps. And rent the French Connection. The chase scene
covers most of Manhattan."
-- Ken Tilton

[Actually it was shot in Brooklyn -- rc]
%
Your other mistake is thinking I was planning a 500-page treatise
complete with suggested legal forms. I am an American, we have Cliff
Notes for haiku.
-- Ken Tilton
%
I will be honest here, I am just a humble application programmer, and as an American I barely know where Monte Carlo is because we only need to know where is Las Vegas.
-- Ken Tilton
%
> "If Lisp is so great why don't libraries, etc. exist for it like
> they do for Ruby, Python, ...".

You are wondering why Shakespeare never conceived a hit game show like
Deal Or No Deal, and why Tiger Woods sucks at miniature golf. Why
Pavarotti never has and never will make it to the Billboard Pop 100,
and why Dale Earnhart got fired after one week driving a taxi in NYC
(he could only make left turns).
-- Ken Tilton
%
The first time you run into something is only the first time you will run into it.
-- Ken Tilton
%
Then again, I do not see why you even want to see it, you already have that sanctimonious glow from using free as in how we defined it software.
-- Ken Tilton
%
What I saw was a defense of Java as being halfway to Lisp and the bit about him having a chart trying to close all the possible holes where behavior was unspecified. True Lispers laugh in the face of unspecified. Hell, we pay extra for it.
-- Ken Tilton
%
Meanwhile Steele famously claims Java is halfway to Lisp. Perhaps he
meant starting from the stone axe?
-- Ken Tilton
%
Apparently these geniuses think it matters one whit whether the spec
in its entirety can be carved on the head of a pin.
--Ken Tilton
%
Now there's a language design principle. Another good one is that you
can sing it to the tune of Camptown Races.
-- Ken Tilton
%
"...Those who can't do, teach. Badly. I am reminded of the New Math,
which made Principia Mathematica (?) a first-grade textbook."
-- Ken Tilton
%
".... Never heard from again, though rumor has it Steele found work
as a tech writer for Sun."
-- Ken Tilton
%
[on motivation to finish his product]

Not to worry, I have a friendly letter here from the IRS asking when
they might expect to see last year's taxes, those tend to focus the
mind wonderfully as well. :)
-- Ken Tilton
%
wants to follow The One True Lisp Way and trust us to know what we are
doing, so compliance here would be compliance for it's own sake.

I must need a drink, that last word looks like a Japanese malt brew.
-- Ken Tilton
%
Had you responded here instead of by email I could have eviscerated you in public (the only thing I really enjoy, the only thing that sets me apart from serial killers).
-- Ken Tilton
%

> And thank you all, (every #'primep ...) worked great. I found it is
> a very nice and helpful place here!

Just wait until you put a parens on its own line, the honeymoon will
be over fast.
-- Ken Tilton
%
> I try never to memorize what I can just look up.

Right. I never memorized C precedence, I dog-eared that one page in
K&R and/or threw in a pair of air-bag parens and skipped the lookup
altogether,
-- Ken Tilton
%
But that code is quite solid and close to Deeply Correct. I know because it has not changed much in years and handles new requirements effortlessly, generally by /taking out/ code that was enforcing disciplines which turned out not to be necessary (and in Lisp we hold inalienable the right to shoot off our own toes).
-- Ken Tilton
%
In the end I remembered that I have never let a concern for accuracy
get in the way of my rants, way too much work.
-- Ken Tilton
%
hard-charging newbies such as yourself landing in Lispville dumbfounded by all the dust, cobwebs, rust, and neglect giving the boot to the war-weary, disheartened, parentheses-mocked old soldiers rolling up your sleeves and setting about dragging the damn language out of the seventies and into the 21st century just in time for the asteroid to hit. What was the question?
-- Ken Tilton
%
You want there to be a problem, just like the strong static typers
want there to be a problem. Unfortunately for all you finger shaking,
rule making, strait jacket wearing school marms we have a nonexistence
proof of craploads of great code being written without a problem in
spite of your sky is falling obsessive compulsive gnashing of the
teeth.
-- Ken Tilton

%
>> Wow, that is two non-required requirements in a week. Me, I am
>> looking for a transmission that can go from forward to reverse at
>> fifty miles an hour without self-destructing. I don't have a need
>> for this, I am just looking for it.
> Easily done. Not so easy is to allow any human passengers to
> survive the event.

Reminds me of the guy I met who said he and his buddy agreed at
sixty-five miles an hour to find out what would happen if they applied
the parking brake. Let's just say it is a good thing that they had
agreed on it, and that the rental car company did not ask how their
car ended up upside down.
-- Ken Tilton
%
If you have to use so many big words, you must be wrong.
--Ken Tilton
%
Please follow up, I want to see if my killfile is working.
--Ken Tilton
%
Yeah, yeah, it was just a rant, you never want those held back by
concerns over accuracy. The sexp/mexp thing esp. suggests divine
inspiration might be a better model than alien arrival.
-- Ken Tilton
%
Unlike the inability of a deliberate mention of Hitler to function as
would an emotionally honest invocation of same to signal the end of a
flamewar, one can apparently climb up on top of the nearest car hood
and announce one is starting a flamewar just to irritate people and a
crowd will immediately form to argue with one over doing so.

I once saw a nature special in which some insect or other dragged some other dead insect somewhere then turned around and dug a whole for it to bury it but the researchers moved the dead insect a bit while it was digging so it had to drag it back but while it did they filled in the hole and back and forth this insect went indefinitely until a PETA sniper took out the researchers. Where was I?
-- Ken Tilton
%
>>I realize other people prefer other environments, they are just
>>mistaken. My ideal setup happens to be the best, hands down.

Wow, I am really out on a limb there. It would be pretty easy to take me down by naming a superior or even near equal environment. Or you could back down in the face of my confidence and resort to, I don't know, name-calling?

> You're clearly deluded.

Understood.
-- Ken Tilton
%
I am not, really. I will do a year or two of Algebra and then have
enough money to open a bar, do what I really want, tend bar.
-- Ken Tilton
%
Mind you I /have/ AG, but there is no point in doing cells-rdf if the
rest of you food-stamp licking, government cheese eating, thrift
shopping oooh-its-gotta-be-free pikers won't be able to benefit from
my unceasing thankless toil on your behalf.
-- Ken Tilton
%
> Cells or Cello might be the solution. But getting mad at people won't
> help.

Thanks for your concern!

Not to worry, I abuse these yobbos for fun, not out of anger. And I
am not sure c.l.l would know what to do with KGK (Kindler Gentler
Kenny), but...
-- Ken Tilton
%
Otherwise, sorry Karl, we are discussing the best way to learn Lisp
(free ACL trial on win32), not the best way to resurrect your six-feet
under commie pinko socioeconomic theory.
-- Ken Tilton
%
> Can someone with a bigger brain than me please elucidate.

I think the problem is not brain-size so much as your admirable attempt to understand a tool by reading about it. That puts you at the mercy of technical writers, who combine an inability to program at all well with an inability to write.
-- Ken Tilton
%
> check this out buddies

I have a buddy?!!! woo-hoo! I thought I had alienated everyone with
my sarcasm!!!!
-- Ken Tilton
%
Tell us more about your home planet.
-- Ken Tilton
%
I'm gonna hurl. Come on, everyone, newsgroup hug....
-- Ken Tilton
%
(a) You seem to be unaware of the Laws of Conservation of Hyphen
Momentum: according to the CLHS, a term in hyphen motion tends to
remain in motion, a term at hyphen rest tends to remain at rest.

(b) Really, the worst thing you can do in CL is use a macro where a
function would do. The Pope does not sudo ex cathedra to say, "It's
not the heat, it's the humidity."
-- Ken Tilton
%
Was the author writing under the cloak of infallibility, channeling
the word of G*d, and is our Talmudic interpretation of those awkward
words precise? The legislative history shows that CL got designed to
address concerns of The Big Customer over language fragmentation, so
it is hard to imagine an intent other than to define one language.
The next step was a pretty tight ANSI standard language specification.

Too easy? (I know, it is more fun being contrarian.)
-- Ken Tilton
%
The language lawyers cannot save you. I am your only hope. I am a
simple application programmer.
-- Ken Tilton
%
Fine, bring me a single malt, a pint of amber back, a wedge of cheddar and some saltines. And we'll need more napkins before we're done with this design.

ps. Oh, and another jar of dijon, and ask the redhead under the moose head if she would like to join us.
-- Ken Tilton
%
> What is a `hacker', or `programmer', or `computer scientist'?

The last two were dragged to their death in the last thread. Hacker is a term used by us computer geeks in a desperate attempt to glamorize our bit-ridden asses, as if the best of us will ever get laid as often as the tone-deaf, rhythym-blind bassist of a third rate cover band on Long Island, let alone the rock stars we pose as when we call ourselves hackers. Paul Graham, who I generally greatly admire and hope will because I said that fund my start-up but more lavishly than he does those Y-Combinator conscripts, drove a stake through the heart of the term here: [snip]
-- Ken Tilton
%
I actually had a business card that just said "Programmer". Got everyone quite upset, they wanted "Systems Analyst" or "Software Engineer" or "Database Administrator" or something. My point was that one cannot program a computer effectively without doing all those things, so "Programmer" was sufficient.
-- Ken Tilton
%
Now can we get back to name-calling? Stop trying to civilize this
brawl.
-- Ken Tilton
%
>> I always tell youngsters it is OK to take one year off before grad
>> school, but for the love of god don't take two.
> I think this depends on the person.

Never look a gift joke in the mouth.
-- Ken Tilton
%
> My theory is that is we bought and open-sourced [...] we could get
> the community to rally around that one,....

The idea of this Lisp community "rallying" is about as conceivable as
a hootenany down at the cemetery.
-- Ken Tilton
%
It is the mouse that feels bad, not the cat playing with it.
-- Ken Tilton
%
The big mistake is thinking Lisp is going to grow by first being
adopted in Tall Buildings. They are the drones, the lemmings, the
sheep. They follow where We the Blessed Gurus lead them. But this
time it is to the slaughterhouse, because the world needs only fifty
Lisp programmers to write All the Code.
-- Ken Tilton
%
... listening instead of yapping? Read my frickin lips: I am talking
about actual tall-office design reviews in which well-paid engineers
... pragmatically suggested design disasters because they objected to
anything a pet rock (sorry, Rockie) could not code.
-- Ken Tilton
%
Anyway, if one has not programmed heads down for three years one likely does not know much about design. I am sure I write more code in a year than academics write in a lifetime, because we are doing different things. Hell, they have the sorry task of trying to pretend there /is/ such a thing as computer science. If there was, wouldn't everyone be using Lisp?
-- Ken Tilton
%
Unfortunately I still do not understand the question, and I am a
frickin genius, I have a three-digit IQ, 50% more than 2.
-- Ken Tilton
%
Well the trick of folks like H.. is to listen just enough so that one can respond to a direct hit with multiple non-pointing counterpoints, each more retarded than the last and each stated in an artfully needling fashion guaranteed to make the sanest NG denizen continue the thread, as if the /next/ direct hit will achieve any more than the last.

It's like playing paintball with a guy who keeps running around and
shooting, covered head to toe in paint.
-- Ken Tilton
%
> deal more. But after seeing your behaviour in cll you can be
> assured that I will never, ever consider any functional programming
> language. I can just do without languages that attract that kind of
> behaviour.

-- tim (and yes, before you respond, that is one reason I
don't use CL so much any more as well.)

Abandoning something wonderful because of who else uses it makes
perfect sense. Why did I give up sex? One word: Joey Buttafucco.
-- Ken Tilton
%
Never been to a code review, have you? You are blessed. The worst crap in the world gets protected by the manager because he is the only person in the room more clueless than the author of the crap and dies when the author dies because the author is in effect a buoyancy device for the in effect non-swimmer manager. But I digress.
-- Ken Tilton
%
I cannot write correct code to save my life, I just throw out any bad code. Only trick there is to distract the author with a banana while deleting their code.
-- Ken Tilton
%
You know, the oil companies have developed a car that runs on carbon dioxide and has like 800 horsepower. Where you can buy one is another question.
-- Ken Tilton
%
Me, I saw the "Microsoft Research" oxymoron and did not get much
further. Unless by research they mean using Google to find out what
ideas other people have successfuly developed and commercialized so
they can copy it badly and crush them.
-- Ken Tilton
%
> Yawn. You must be a riot at parties.

And you must be the life of a funeral.
-- Ken Tilton
%
The way to get this going is to post here an especially good RQ
question and your Lisp solution, see if you can drum up interest. If
it takes off, you start a Web site or something. If not, the ball
game comes on at two.
-- Ken Tilton
%
ps. I agree, the "fingers will be chopped off" sign should not have
been Comic Sans. :)
-- Ken Tilton
%
Well whaddya know. I do GUIs, leave file work to the chimps. You win
a banana.
-- Ken Tilton
%
Can we continue this over on comp.lang.turing.complete?
-- Ken Tilton
%
...the only thing that matters is Becoming the Latest Thing. What is
the latest thing? The (a) new thing (b) being recommended (c) by
Famous People (d) in respectable places
-- Ken Tilton
%
The operator helpfully suggests that I could avoid this problem by simply saying that my mother's maiden name is YOBBO, no one will make fun of me. He also takes care of confirming the purchase over the phone, while I try to figure out how to sell "Hi, I need to change my mother's maiden name..." to the next operator.
-- Ken Tilton
%
So I am working in a lab? That would explain all the beakers.
-- Ken Tilton
%
That rose girl is drying up and a few volcanos need sweeping.
-- Ken Tilton
%
Grapette this: no, Jodie Foster is /not/ responsible for John Hinckley
shooting three people including the President. The correct question
was "Who is John Hinckley?".

Really, kiddies, it is OK to blame the perpetrator. Not that I do in
this case. The OP is clearly an unhappy puppy deserving nothing other
than compassion for the demons that drive them to randomly attack
Usenet blowhards like me.
-- Ken Tilton
%
And Uhl says Kenny has made the South forget the Civil War, not sure
how I could top that. The hounds are exhausted, smiling in their
sleep. It's all good -- but someone has to talk Bubba and Jethro down
from their sniper nests.
-- Ken Tilton
%
Do they have the distinction between a priori and a posteriori in your
banjo-pickin, moonshine-slurping, carcinogen-growing,
basketball-playing neck of the woods? How about de jure vs. de facto?
Simular.
-- Ken Tilton
%
That /was/ a despicable and wholly unjustified ad stateum low-blow, a
cheap shot deliberately designed to make me look bad. It crosses a
shocking line, beneath contempt, really. Clearly I had cut in my
flamethrower after-booster and lost all reason or sense of decency.
It almost makes me wonder if I was even serious...
-- Ken Tilton
%
See, this is what happens when you get all riled up and change the
subject to whether Kenny is consistent or not, you get so emotional
you cannot read straight. And my inconsistency varies quite a bit, so
I do not know if you can build an argument on that anyway.

ps. Random problem cloning is going well, but obviously slowly enough
to have me dashing here to hide pretty regularly. :) What kind of Lisp
did you write today?
-- Ken Tilton
%
MWUAHAHAHAHAHHAHAA!!!!! Yes, palindromeloopstate set to 1 makes the
movie run endlessly forth and back. Picture looks fine running
backwards, but instead of the spoken words coming out in reverse,
well, it is just unrecognizable noise. Working on that now...
-- Ken Tilton
%
Fine, but anyone who uses Usenet knows that that is not how one says
"Thanks". Which is why I felt safe moving directly to defcon 3, just
to see how big an *sshole we had on board. His defcon 1 "Rot in hell
jackass" response merely, well, QED.
-- Ken Tilton
%
[...] the serious answer comes from the Tao Te Ching, lessee, on about
every other page:

"A man on tiptoe cannot walk easily."

Or:
"Never trying to impress, their being shines forth Never
saying 'this is it', people see what the truth is -- Never
boasting, they leave the space in which they can be valued ...
And since they never argue, no one argues with them either."

Lao Tzu clearly would be no fun in a flame war.
-- Ken Tilton
%
"Gotcha" is like analogies, they lead to arguments about the
gotchaness of the gotcha, an exponential explosion guaranteed.
-- Ken Tilton
%
Despite copious opinion to the contrary, I speak not as Grand Poobah
of Common Lisp and have not the power to marshall our forces to
maximize the benefit of their labor. I can only pause between rides
to town on my pushbox to wonder aloud in camp why a serous chunk of
our tribe is over there under the tree working on the wheel and
axle -- ones no better than the one on my pushcart.

Kenzo: "Dudes, it's been twenty-five years, prices down at Throg's
Wheel & Axle aren't all that bad. Why not work on a mast and sail?"
-- Ken Tilton
%
A subtle execution of the tip of a tongue pressed against the upper
teeth with sprays of spittle coming out either side probably is not
what you had in mind.

Hmmm. Then we change the spelling to Lithp, and never have to hear
that stupid joke again. Our slogan can be "Thay it loud, thay it
proud."*, and we already have the frickin lambda.

Or "Out With Lithp!".
-- Ken Tilton
%
Barker: "Stairway to Heaven! Open to all! Come on up! One-day
amnesty!"

Kenny: "No elevator?"
-- Ken Tilton
%
No, I made a few changes and sent it to the Copyright Office.

For Mom I am holding out for the cover of Wired. (She already takes
America's Most Wanted.)
-- Ken Tilton
%
"you also physically incapable of understanding that your opinions are
not the words of God". Nice comeback on originality! Physical
understanding? What part of the mind-body problem do you not
understand?
-- Ken Tilton
%
No, I would mock you for being so locked into mob-rule and aggression that you think my insistence on walking to a different drummer entails also abusing anyone not conforming to my non-conformity.
-- Ken Tilton
%
"wet feet" might not do justice to the 3D learning curve -- that
subsonic rumble shaking your kayak is Niagara Falls.
-- Ken Tilton
%
"Feeling no pain, Kevin?"
"Sorry?"
"You just came out of the women's rest room."
"Look, I had to take a leak. Odd, no urinals, just sit-downs."> & no
'bouncers' made any approach..

It's crowded, they are still working their way through the crowd.
Don't worry, we'll explain about the Lisp "high".
-- Ken Tilton
%
er, um... no. OO is about managing huge wadges of code in huge
systems that will be spending most of their lives waiting on disk I/O
so WTF cares about OO overhead, we need to manage these huge
codebases!!! ie, OO is for lazy-ass mo-fos who cannot be bothered to
toss of a few dozen lines of code and reinvent "objects" in a fashion
screamingly optimized for their application.
-- Ken Tilton
%
So it samples the pitch and guesses at rhythm and provides tutoring
similar to what I would get from a good music teacher? Astonishing.
AI has been solved! Stop the presses on the emasculation of George
Bush, we have real news!!!
-- Ken Tilton
%

> if Clisp is so good where are the commonly used apps?

If George and Barbara had such great sex, how did they produce Jeb and Dubbya?
-- Ken Tilton
%
> I like the lizard.

No, you don't, you are just saying that. Think again.

> But then, I like the Geico gecko, too.

Everybody likes the GG. It has an australian accent.

> And I like Common Lisp, too. Guess I'm weird, hunh? ;-}

I was thinking "doomed".
-- Ken Tilton
%
And a mascot like Joe Camel to suck in the kiddies, gotta have a
mascot.
- Ken Tilton
%
You yobs might want to ... oh, what's the use? Lisp is dying. The next generation of Lispniks is over on #lisp worshipping themselves and learning nothing and achieving less. My god, another two years and Slime may be half the power of the ACL IDE. What is the word I am looking for... ah, here it is: PFFFFFFFT!
-- Ken Tilton
%
You French really are pissed off about Lance Armstrong, aren't you?
-- Ken Tilton
%
* reminds me of part of a route description to a rock climb called
Death's Door: "Don't use the jug handle just to the right of the
finger jam, that hold is part of Cakewalk."
-- Ken Tilton
%
I have worked with body shop programmers who could not be bothered to write structured code. Are the concepts of structured programming too hard? Nah, those people just "refused to be bothered" (a direct quote), meaning they were too inured to the pain of spaghetti coding to realize how much "bother" structured programming could save them. They thought spaghetti code was /easier/ because, hey, how hard does one have to think to add another GOTO? It breaks somewhere else? Add another GOTO! C'mon, this is easy! Breaks somewhere else...read my lips: GOTO!
-- Ken Tilton
%

The King is Dead!? Long live... Scala? Clojure?!

2009-10-31T13:11:00.004-04:00

Whoa, why wasn't I told Java is closing its doors? I guess I have been out of touch, word seems to be everywhere. I had to go here (blog of the guy who created Groovy) to find out. James is whooping it up over Scala as Java's successor. Yes, the guy who invented Groovy prefers Scala. Quite a bit:

I can honestly say if someone had shown me the Programming in Scala book by by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I'd probably have never created Groovy. -- James Strachan

Damn. So he pretty much invented Groovy by mistake? Did not know about the two-year old Scala? No one mentioned it to him? Groovy got admitted to the standard in the meantime?

Well, Johnathan Edwards reinvented Python Trellis (ergo Cells) without knowing it, and I did not know about Garnet's KR or constraints -- but Groovy got adopted as official Java! You think Scala might have come up over coffee. Anyway...

Steele said Java brought the world half-way to Lisp. I do not think Lisp means what he thinks it means. Proof might be how hard it is for folks to climb out of the pit of javathink. If Java had been a stepping stone to Lisp it would have made the next step easier, not harder. But Java still cannot do closures. Please. And a quick look at closures in Scala has me thinking, omigod, they call that closures?

Clojure starts to look like a Good Move. I see it mentioned in writings on the death throes of Java and that is a big marketing win. The superwhacky thing here is that both Scala and Clojure are syntactically discontinuous from Java. Folks always thought successors had to have syntax similar to the succeeded though Dylan should have served as cautionary counter-evidence.

No, it is not the syntax. The necessary bridging element seems to be....wait for it...the Java runtime! How did that tail end up wagging the language adoption dog? But Clojure gets the nod along with Scala just for sitting atop the JRE! You people scare me.

Well, if Steele were right Clojure would prevail over Scala. Right now googlefight has Scala winning five to one. Maybe Rich Hickey can move 17% of the world 90% of the way to Lisp?

To write, or not to write?

2009-08-10T06:01:00.000-04:00

>> By the way, to change the subject a little, who was it who said,

>> "there are no dead languages, only dead minds"?

> Dunno, but to change the subject even more, Socrates objected to writing

> since it deprives an idea of a mind in which it can "live". So yeah.

Interesting. "Free writing" is a form that lives within a mind but also create a permanent record and slow the mind down enough to achieve more coherence so the mind can work out hard problems. Comedy writing is necessary to trigger a laugh response because every word matters, but then the words must be delivered as if they were coming live from a mind. Exceptions are improv and semi-improv such as Eddie Izzard, of which Mr. Socrates would approve because they arise within a living mind.

I get a lot of complaints about the writing in this blog because I deliberately write as chaotically as I think. Other times I found I gave a much better talk if I read from something written beforehand precisely because otherwise the living mind is too chaotic to get the talk done in anywhere near the time available.

I was just getting ready to videotape an improvised bit to get the good bits to then pull into a fixed, written bit because I am finding good stuff comes out only if the mind is not slowed down as by free writing.

The question is whether Eddie Izzard is lazy, or if Socrates is right on this. Does Izzard do better by capturing his improv and distilling it down to a precise fixed bit, or does he do worse? Or does he just lack the ability to deliver the prepared as if it were unprepared.

We're getting pretty close to talking about programming in Lisp vs NotLisp now. Lisp programming unconstrained by static typing and blessed with a rich library once that library is mastered such that it is all at the programmer's fingertips allows the code to flow freely yet mostly correctly from a live mind even as that mind is forming the solution the code embodies. Diagram that.

I Feel A Naggum (RIP) Coming On: Quads

2009-06-26T00:47:00.007-04:00

I sometimes begin c.l.l rants with "I feel a naggum coming on...". What is a naggum? Normally:

naggum (n): A rant along one of Erik Naggum(1965-2009)'s themes.

That might be self-referentially hopeless which is fine because that is not what I am talking about, I just thought that would be a clever title. In this case a "naggum" is a nugget of Erikian technology. First, his specification of what he called quads (see below), and my poor implementation (even further below and good luck even figuring out how to test it) .

From: Erik Naggum (erik@naggum.no)

Subject: Re: XML->sexpr ideas

Newsgroups: comp.lang.lisp

Date: 2004-01-19 04:24:43 PST

* Kenny Tilton

| Of course it is easy enough for me to come up with a sexpr format off

| the top of my head, but I seem to recall someone (Erik? Tim? Other?)

| saying they had done some work on a formal approach to an alternative

| to XML/HTML/whatever.

| True that? If so, I am all ears.

Really? You are? Maybe I didn't survive 2003 and this is some Hell

where people have to do eternal penance, and now I get to do SGML all

over again.

Much processing of SGML-like data appears to be stream-like and will

therefore appear to be equivalent to an in-order traversal of a tree,

which can therefore be represented with cons cells while the traverser

maintains its own backward links elsewhere, but this is misleading.

The amount of work and memory required to maintain the proper backward

links and to make the right decisions is found in real applications to

balloon and to cause random hacks; the query languages reflect this

complexity. Ease of access to the parent element is crucial to the

decision-making process, so if one wants to use a simple list to keep

track of this, the most natural thing is to create a list of the

element type, the parent, and the contents, such that each element has

the form (type parent . contents), but this has the annoying property

that moving from a particular element to the next can only be done by

remembering the position of the current element in a list, just as one

cannot move to the next element in a list unless you keep the cons

cell around. However, the whole point of this exercise is to be able

to keep only one pointer around. So the contents of an element must

have the form (type parent contents . tail) if it has element contents

or simply a list of objects, or just the object if simple enough.

Example: 123 would thus be represented by (foo nil "123"),

123456 by (foo nil "123" bar nil "456"), and

123456 by #1=(zot nil (foo #1# "123"

bar #1# "456")).

Navigation inside this kind of structure is easy: When the contents in

CADDR is exhausted, the CDDDR is the next element, or if NIL, we have

exhausted the contents of the parent and move up to the CADR and look

for its next element, etc. All the important edges of the containers

that make up the *ML document are easily detectible and the operations

that are usually found at the edges are normally tied to the element

type (or as modified by its parents), are easily computable. However,

using a list for this is cumbersome, so I cooked up the «quad». The

«quad» is devoid of any intrinsic meaning because it is intended to be

a general data structure, so I looked for the best meaningless names

for the slots/accessors, and decided on QAR, QBR, QCR, and QDR. The

quad points to the element type (like the operator in a sexpr) in the

QAR, the parent (or back) quad in the QBR, the contents of the element

in the QCR, and the usual pointer to the next quad in the QDR.

Since the intent with this model is to «load» SGML/XML/SALT documents

into memory, one important issue is how to represent long stretches of

character content or binary content. The quad can easily be used to

represent a (sequence of) entity fragments, with the source in QAR,

the start position in QBR, and the end position in QCR, thereby using

a minimum of memory for the contents. Since very large documents are

intended to be loaded into memory, this property is central to the

ability to search only selected elements for their contents -- most

searching processors today parse the entire entity structure and do

very little to maintain the parsed element structure.

Speaking of memory, one simple and efficient way to implement the quad

on systems that lack the ability to add native types without overhead,

is to use a two-dimensional array with a second dimension of 4 and let

quad pointers be integers, which is friendly to garbage collection and

is unambiguous when the quad is used in the way explained above.

Maybe I'll talk about SALT some other day.

Erik Naggum | Oslo, Norway

Act from reason, and failure makes you rethink and study harder.

Act from faith, and failure makes you blame someone and push harder.

(in-package :ukt)

;;;(defstruct (juad jar jbr jcr jdr)

(defun qar (q) (car q))

(defun (setf qar) (v q) (setf (car q) v))

(defun qbr (q) (cadr q))

(defun (setf qbr) (v q) (setf (cadr q) v))

(defun qcr (q) (caddr q))

(defun (setf qcr) (v q) (setf (caddr q) v))

(defun qdr (q) (cdddr q))

(defun (setf qdr) (v q) (setf (cdddr q) v))

(defun sub-quads (q)

(loop for childq on (qcr q) by #'qdr

collecting childq))

(defun sub-quads-do (q fn)

(loop for childq on (qcr q) by #'qdr

do (funcall fn childq)))

(defun quad-traverse (q fn &optional (depth 0))

(funcall fn q depth)

(sub-quads-do q

(lambda (subq)

(quad-traverse subq fn (1+ depth)))))

(defun quad (operator parent contents next)

(list operator parent contents next))

(defun quad* (operator parent contents next)

(list operator parent contents next))

(defun qups (q)

(loop for up = (qbr q) then (qbr up)

unless up do (loop-finish)

collecting up))

(defun quad-tree (q)

(list* (qar q)

(loop for childq on (qcr q) by #'qdr

while childq

collecting (quad-tree childq))))

(defun tree-quad (tree &optional parent)

(let* ((q (quad (car tree) parent nil nil))

(kids (loop for k in (cdr tree)

collecting (tree-quad k q))))

(loop for (k n) on kids

do (setf (qdr k) n))

(setf (qcr q) (car kids))

q))

#+test

(test-qt)

(defun test-qt ()

(print (quad-tree #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))))

(print #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))

#+xxxx

(test-tq)

(defun test-tq ()

(let ((*print-circle* t)

(tree '(zot (foo ("123")) (bar (ding) (dong)))))

(assert (equal tree (quad-tree (tree-quad tree))))))

(defun testq ()

(let ((*print-circle* t))

(let ((q #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789"))))))

(print '(traverse showing each type and data preceded by its depth))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)(qcr q)))))

(print `(listify same ,(quad-tree q))))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

;;;(defun tree-quad (tree)

(defun testq2 ()

(let ((*print-circle* t))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

American & Iran: Separated at Birth?

2009-06-22T01:16:00.002-04:00

Am I the only one grooving specifically on the fact that Iranians are telling their authority figures to go f*ck themselves? Here is the country we thought we hated but it turns out they are as kick-ass as us when it comes to political freedom, and we utterly respect them for their strength. Omigod, Americans and Iranians are going to get along great!

How to teach math

2009-05-11T00:34:00.006-04:00

> On Sat, 09 May 2009 15:30:52 -0400, Kenneth Tilton wrote:

Well, i was not really trolling, I was forking the thread to make fun of the New Math that tried to get the numeral/number distinction across to five year olds.

Someone responded:

You find it better to start with medieval concepts working gradually on to the mathematics of XIX century, while explaining each next year what was wrong with the things they learnt a year ago?

I said all that? Ma's gonna be right proud. But...

Funny you should ask. Yes, I suspect the path society took to get to what it knows now about math is the path an individual neuronal mass should follow. ie, kids should encounter zero and roman numerals and place value and algebraic variables in the same order society developed those ideas. The history of math is your math curriculum guide.

The New Math erred by selecting the logical organization of mathematical concepts as its curricular pole star. Next came Constructivism, which wanted kids to reinvent math. From scratch. Cool idea, but too slow.

Instead, let the history of mathematics dictate the order in which things are directly taught. Maybe go further and teach math as history with less emphasis on computation. Math often advanced when needed to solve real problems. Maybe we can shut up the little devils asking why they need to learn this stuff.

As for explaining all along the way what was wrong with the ideas taught the day before, hey, ever read a book on programming? They typically develop a chunk of code iteratively, presenting ever more improved variations on a primitive original. Come to think of it, ever develop some software? Same thing.

Here's the deal: most folks do not even know zero had to be invented. One understands zero better if one has done without it and then the teacher invents it for you. Something like that.

Cells: The Secret Transcript

2009-02-04T11:10:00.002-05:00

The Boss asked me to give the group fifteen minutes on Cells because I have been talking about it for a while as a future better mousetrap for us and then suddenly last week threatened actually to apply it to qooxdoo and the front end.

I forwarded to everyone a link to a reasonably complete yet relatively brief write-up which tells you everything you need to know about Cells. I know that if I were in your shoes I would not have read it so I presume no one has. But I would like to determine how many folks I will be boring to tears if I review said document, so I will first cut to the chase and ask if anyone has any questions based on what they read.

........silence..............

OK. Cells is at once the simplest and hardest thing in the world for programmers to understand. Simple because the idea is just to have slot values of objects work like cells in a spreadsheet, and everyone knows how spreadsheets work. What is hard is understanding that one can program computers this way.

I only have fifteen minutes so: Yes, you can. Program inputs are assigned by good old imperative code to input cells the same way a user types values into a spreadsheet when they are doing what-if analysis or recording, say, actual monthly expenditure into a budget spreadsheet. Intermediate, derived, and aggregate cells compute new values based on those new inputs from predefined rules just as user changes to a spreadsheet propagate to other spreadsheet cells. Observers on cells let the emergent working model manifest its decisions with more good old fashioned imperative code, usually by simply updating the screen or playing a sound or controlling some external device over a serial port or updating a database.

Boom, we're done. Why is it so great? What part of the superiority of functional and declarative paradigms should I explain first? As I outlined in the material you did not read, a lot of things have to happen when a program receives an input. The programmer coding the event handler has to look at the event and decide all the things that have to happen in light of that event, and any things that follow from those first things. Not only must they reliably see to all those things, but they must do them in the right order. The analogy to a real spreadsheet is quite strong, if you imagine hand-implementing a spreadsheet with old-fashioned pencil and paper.

So the first things Cells does is eliminate a lot of work and thus a lot of bugs. Because the work eliminated is tedious, Cells also makes programming a lot more fun. But there is more.

The declarative paradigm means I always know why a slot has a certain value, because all the logic appears in one place, in the rule assigned to that slot. Without Cells any number of lines of code may have assigned a value to a particular slot and a unified deriving rule certainly cannot be divined even if one were to track them all down.

There is more. Most people hate OO because it never quite panned out. Objects turned out not to be reusable. One of the nicest features of Cells is that two different instances can have different rules for the same slot. That makes objects reusable. Yayyyyyyy.

I did not use Cells in the Kleaner because that was more of a straight calculation running from start to finish. I like to decribe the role for Cells being in any situation where one has an unpredictable stream of data and one is keeping a model with a sufficiently large amount of internal state consistent with that stream of inputs. Two examples being a GUI and a RoboCup client. The Kleaner worked by compiling statistics from a fixed store and then translating exactly once dirty data into corrected data.

Where Cells would have been useful would have been in implementing Phil's ideas about an ongoing stream of data leading to rediscernment of things previously discerned. Cell rules would take new raw inputs and propagate them over to tables of probabilities which would then reach out to existing cleaned data and possibly redecide from the original raw state a new cleaned state.

So no, I do not use Cells for everything, but in this case it was only because the full functionality had not been addressed.

A fun note is that I have in the past applied Cells to a database, specifically the old AllegroStore persistent CLOS database. This works two ways. One is that a user can be looking at a screen and as the underlying data changes the screen changes. That may sound like old news but with Cells one doe not have to write any code to make it happen. One just says "this view shows this users overdue books" and when the date changes the overdue status on every book gets updated and a new book appears in the list on the screen if someone happens to be looking, simply by someone having written code to list overdue books on the screen as if it were an unchanging value.

The other thing that happens is hinted at above. Things like overdue books and amount of fines owed and paid can be calculated from scratch by reading a users entire history of checkouts and returns, but sometimes it is useful to record such derived values in the database and update them incrementally as books are checked out and returned. We can have code in programs do it and hope they run at the right time, or we can have the code in the database (as datapoints mediated by Cells) and be sure they run and run immediately. We get timeliness of data, efficiency, and we still get consistency even as we introduce redundancy.

I left something out. That is bad because the thing I left out is the thing I am planning to do with Cells on my own anyway and then apply to the FE. Cells makes it dead easy to drive a separate framework from Lisp. In my note I mentioned tcl/Tk and Gtk. These are two killer C GUI frameworks with their own homebrewed little object models. We want to program in Lisp, and we want our models driven by Cells for all the reasons above. No problem. We build a model out of instances of CLOS classes mapping isomorphically onto Tk or GTk classes and use Cell observers to pipe information (thru an FFI or even literally a pipe) to the C library or runtime to drive there the creation and animation of C instances.

Works great, and one amazing programmer Peter Hildebrandt pulled off a trifecta in which he had Cells driving and driven by both GTk and a C physics engine, name forgotten.

For a while I kinda marvelled at how Cells could be so useful for such disparate activities, and do so in the same application, the two activities being building an application model and having some other programming framework dance to that model's tune.

I figured it out in time for ECLM 2008, not they were able to understand me. I opened by telling them that Cells was the single most powerful library they could use, because Cells is about change and nothing is more fundamental than change.

Programming is hard because like someone doing a spreadsheet on paper we programmers end up with the burden of propagating change thoughout our models. It is tedious work, it must be done reliably, and there is a lot of it as internal program state multiplies, exponentially a lot. This exponential growth in interdependence of program state is what led Brooks to declare that a silver bullet was not only unlikely to be found but that it would be impossible to find; he felt the complexity was ineluctable because as states multiply there is nothing that can be done to avoid the explosion of interdependence.

I wrote to Dr Brooks recently and asked him if he had ever looked at dataflow. He said he was familiar with the concept, but no. Oops.

Tinkering

2009-01-25T15:12:00.001-05:00

Spring cannot come soon enough for Bobi, who is losing it badly these days on comp.lang.lisp:

Slobodan Blazeski wrote:

Dear board members

I'm baseball player for a several time periods (days,
moths ,years,decades) I've noticed that interest in baseball is
dwindling, and baseball is becoming less and less relevant and will
soon become extinct with only baby boomers supporting it, and even
those are either going to die or switch to golf. In order to save our
favorite sport I propose we make drastic changes and adapt more modern
things like:
a. Playing on the beach sand wearing swimwear like in beach
volleyball, very modern sport. Check Thiobe for growth rate
b. Replacing bats with hockey sticks. Note that hockey is popular in
many world countries and we should think international
c. Including 24-Second Shot Clock like in NBA that will make our
sport more lively and fast paced
d. Square playing fields should be replaced with the more common
rectangular one like found in many popular sports : soccer, football,
tennis etc

Including this will make baseball prosper.

very truly yours

Concerned Semi-Ex Baseball Player
Avenue of delusional weirdos Number 23

Bobi may not be as crazy as he thinks he is. Baseball suffered extreme popularity anxiety in the late Sixties and did indeed tinker with the game. Thinking more offense would attract more fans the pitching mound was lowered so pitchers did not get extra energy into the ball from falling into a pitch. The American League adopted the designated hitter to eliminate the 11% nil pitcher from batting lineups (eliminating as well an awful lot of interesting strategy). They avoided the salary caps of the NBA and instituted free agency (well, no, they lost a lawsuit) which allowed bigger markets like NYC, Boston, and LA to buy better teams, and bigger markets are always good for ratings. Minnesota fans will follow the Dodgers, Los Angeles fans will not follow the Twins.

The changes went beyond the playing field. Ballparks added mascots and a disgusting cacophony of party music between innings so loud you can barely talk, and limited alcohol sales late in games to make the experience more family-friendly cuz you know how the losing fans get in their third hour of drinking.

Now baseball is hugely popular again so tinkering with grand institutions can work. Right?

Wrong. In the end, baseball is just a great game: multi-dimensional and deep. Quality tells, and which quality one emphasizes matters. Hockey and basketball have non-stop action and are fading in popularity, while baseball and football like great music have a variety, a rhythm, a balancing of quiet against intense. Baseball has the pitch, football has the snap. All scales from small to large from inning or drive to the game or season always and invariably end up condensed into one point of explosive tension when the pitcher releases or the center snaps the ball.

Intense without quiet merely exhausts. A boxing match with two brawlers spurning defense landing bombs back and forth brings the crowd to its feet but those who love the sport do so for its nickname, The Sweet Science. They still talk about one genius of defense who won a round without throwing a punch. Between evenly matched fighters one solid punch (forget the knockout, the cartoon haymakers of Rocky n) brings the crowd screaming to its feet, the culmination of rounds of careful, tentative, mutual exploration. A single knockdown becomes a cause for pandemonium and one punch knockouts almost do not happen between the best and when they do they are talked about for a long time. I digress.

Tinkering. Basketball has all the action in the world and now faces its own popularity crisis. Racism is one factor, another is probably the salary cap that has San Antonio in the championship series instead of New York. Another problem: poor defense, and a twenty-point lead does not mean anything.

But worst of all is the lack of dimensionality. There just is not that much to these games to argue about over the water cooler. Baseball? Boston still talks about the time Grady Little [thx, Xach. ed.] left Pedro Martinez in one inning too long against the Yankees in game seven of the ALCS. Come on, he had thrown a hundred pitches! Everyone knows Pedro is useless after a hundred pitches! You just never hear anything like that about hockey or basketball, which both boil down to great athletes pretty much just playing run and gun.

Baseball never needed tinkering, though tinker they did. The fundamental quality of the game first ensured its survival throught the hard times when fans strayed for the quick fix of non-stop hockey and basketball action. Now the richness, subtlety, and sophistication of the game has some stadiums selling out most games of the year of a very long season.

Moral for Lisp left as an exercise.

Tilton's Law: Solve the Failure First

2009-01-20T13:38:00.005-05:00

The team was at my throat.

"Just use the new search!," they bellowed.

The mission critical, project saving, do or die demo to upper management was eight hours away and we had not even begun the always dicey process of moving the software from the development system to one within reach of the Demomeister, and I was trying to find out why the old search was so slow.

"Soon," I replied.

We had a new search I was told was a screamer but I continued poking around putting in metrics trying to figure out why the old search was so slow. Had we not been a virtual remote telecommuting team I would not have lived to tell this tale but we were so they had no choice and I reassured then that "soon" meant ten minutes and they shut up.

Why was I still trying to understand the perplexing sloth of the old when a whole new replacement module was available and working fine and pretty much the demo on which all our jobs and a cool project depended was coming on like freight train?

Tilton's Law: Solve the failure first.

Early on we learned the other side of that coin: Solve the first problem. The commonality is...no, let's do the war story first, war stories are more fun than preaching.

Back we go a quarter of a century to my first contract with a client who would become my sole recurring client for the next decade. I was being hired to take over maintenance of an application whose author had been one of the first to die of AIDS. I was reminded of the whole business by a conversation with another developer recently about the nature of working on OPC. Other People's Code.

In my IT career I have worked always at the poles of software development, either writing new code or performing massive overhauls of OPC, never that relaxed zone between in which one simply maintains and extends in small ways a long-lived system. The second pole (OPC overhauls) always seemed to me an intimate one-way encounter with some anonymous predecessor, an encounter usually involving me roundly and steadily cursing them out. You can imagine then how eerie it was working on this system from this predecessor who was not so anonymous this time, especially when I learned that the poor guy was in bad shape during one stint but needed the money and so worked on the code I was now working on even as his fate rose up to meet him (this well before the days of the cocktails of today that make ones fate less certain). This guy I do not remember cursing out so much.

But I digress. Our lesson today is how to piss off your coworkers by insisting on solving a failure first, by which I mean even if you do decide to punt on X make sure you understand how X failed. I am not alone in this. In 2001 the movie when the crew determines that the unit Hal said was no good was fine he says fine let's put it back in and let it fail. Sure, he was really looking for a way to kill the crew but we learned in 2010 that Hal was just a computer system and I think the bit about putting the supposedly OK/not OK system back in to see if it failed was one of Hal's systems working nominally in accordance with Tilton's Law: we need to understand broken things.

[Aside: a good example of the Hell of Not Solving Failure.]

And now at long last, my unsolved failure. My predecessor's, actually. The application was a securities database with a nightly feed of data applied to the cumulative DB by a batch program. This is late 80s, primitive stuff. A security could have three IDs because three groups were tracking securities and each had their own ID system. We had tens of thousands of records in our VAX/VMS RMS file, and a separate RMS key for each of the three possible IDs. So far so yawn. Here comes the fun part.

Two of the IDs were populated all the time. The other one was populated five percent of the time. Big deal, right? Right, very big deal, the poster boy for Solve the Failure First. What happened was this quesswork reconstruction:

My predecessor Paul (I picked "Paul" because it easier to type than predecessor) had a problem. His program ran an initial test load of a hundred securities in a few seconds. Fine. Everything looked good. So then he ran it against a full daily feed, which would include news of every security traded that day so it would be -- OK, I confess I completely forget even the order of magnitude, let's say tens of thousands and declare up front that that is idiotic and I am sorry, but here is what happened: the damn thing ran forever. There probably was no immediate specific great mystery because Paul probably had the program printing something (a count, the last ID recorded, something) right to his VT-100 console as it went and he could see that the program had started out zooming along but then gradually got slower and slower until just adding one security to the database (and this is just good old ISAM, mind you) took... wait for it... twenty seconds. Oh. My. God. What on earth is happening?

Paul got a clue. Every once in a while two records were written out bang-bang, as fast as at the start. Dig dig dig puzzle puzzle...ah, there it is. Any record for which we have all three IDs is written out in nothing flat. Any record (you know, the ninety-five percent) with just two will (by the end of the run) be written out three per minute, 180/hour, or 1000/fuggedaboutit.

Paul realized what was going on. The ISAM file system had no problem storing data with duplicate keys, which was a good thing because Paul was storing a whole lot of data with one key 95% the same: spaces. Poor ISAM it seemed was chugging thru all the duplicates looking for the last one after which it would record the latest duplicate. And apparently it took twenty seconds back then to walk (effectively) the entire index of a hundred-thousand record file.

Now the good news is that we would never need to look something up using spaces as the key value sought, so....what can we do? Paul was no slouch. He popped open the RMS reference manual and to his delight discovered he was not the first to pass this way and gleefully added the option "NULL_VALUE=SPACES" (translated: "if the value is spaces, Just Don't Index this record on this key") to the key definitions in the file definition script he was using to initialize the file and recreated the file and re-ran the program from scratch.

The change did not help. At all. I think we all know that feeling, as visceral as a dropping elevator.

There it was, the option explicitly intended to solve the problem he had explicitly encountered, and it did not change a thing. Impossible. But this happens to us programmers all the time. We know what to do. Compile the damn code, because we made the edit change but forgot to compile. Or link. Or, in Lisp, to zap the faultily specialized method. Or something.

So Paul edited the definition again and checked that NULL_VALUE = SPACES was the right syntax and right spelling and on the right key -- to hell with that, he put it on all three damn keys -- and he saved it and checked the date and created the file again and ran his program again and you know it did not run any faster or I would not be telling this story.

OK, time to get serious. Or if he was good he did this without all the huffing and puffing of the preceding paragraph. He Just Typed In "rms/analyze sdb.dat". And RMS looked at the file itself (not the script used to create it) and confirmed that "NULL_VALUE = SPACES" was operative for all indexes.

Momma don't let your kids grow up to be programmers.

What comes next is hard to convey. I can tell you but if you have not worked on this code or (we will learn) run this batch application it is hard to convey how much blood, sweat, tears, CPU time, and delayed nightly batch closes for how many years resulted from Paul's not first solving the failure of NULL_VALUES=YES.

Well, maybe this is a fair glimpse of the enormity that followed: the problem got sorted out only because the head of operations and I got to talking one day and he pretty much ended up down on one knee begging me to find some way to eliminate the two-hour merge step that held up the nightly close every night. I had never heard this before, though I had been overhauling the system for months.

"It just sits there for two hours," he groaned. "It kills us every night. Please, if you can, please, do something to make this go away."

Whoa. I had inherited this system and been asked to enhance it but no one had said a word about this. The code was far and away the best OPC I had ever dealt with so everything got the benefit of the doubt, including the (soon-to-be explained) two hour merge. As in, if it is there, it must be there for a good reason. What was not there was The Story of the Unsolved Failure of NULL_VALUE=SPACES, but even if it had been I would have taken that at face value, too, because the NULL_VALUE option was unknown to me. But enough of this flash forward, let's get back to poor Paul.

NULL_VALUE was not working as it should. Software is like that. Good programmers do not let bad software stop them. Plan B. A rule is born: Thou shalt not write new securities to the securities database where the massive duplicates will make each write take twenty seconds. Paul decides to write them to a second file initialized empty on each run. Since we only got dozens of new securities in one batch, that file would never have the massive count of duplicates and writes would be lightning fast. Then we just do a sort/merge at the end of the batch to combine the new securities in with the old. Oops. "Just."

The funny "you can run but you cannot hide" moral within the moral being that I did the calculations one day and worked out that twenty seconds times the usual number of new securities in a day was exactly as long as the sort/merge that was just killing the folks down in operations. And I bet Paul realized that but only after writing all the crazy code he had to write to work with two files at once as if there were only one file and at that point he just gave up and moved the thing into production. Speaking of crazy code...

You should have seen it. Looking back I cannot recall why it should have been so hard, but I did overhaul that code and I was forever tripping over it. The idea is simple. To look up a security to see if we already have it, first look in the real DB and if it is not there look in the daily "new stuff" DB and if it is not there, ah, it is new. If it is, update it. Just remember to update the right file, because we can get data from two sources about the same new security.

Piece of cake, right? A bottleneck function for all reads and updates... anyway, it seemed like the issue was always getting underfoot as I worked, and just looking at the code one saw again and again this check here/there code, and both Paul and I were the kind of engineers always on the lookout for ways to make code non-redundant. I would think my memory was faulty but I also remember eliminating Paul's Plan B after solving his failure first and that was no picnic. It just permeated the application.

So what was the first failure and how did it get solved? First, I had noticed the issue myself while using Datatrieve to add a record to the securities DB for test purposes. I hit enter and thought I had crashed the system because it went away and never came back and like every egomaniacal programmer out there I always assumed that whenever a system stopped responding the last thing I had done must have broken it so there I sat in dread for twenty seconds until the system finally responds. Wow. Twenty seconds? And then I guess I added a record specifying all three keys and it responded instantly.

But this idea of null values not being recorded in an index was new to me, and we did not have the Internet back then where I could just ask the ether what was going on so it was only a coincidence that just after the guy in operations had begged me for a fix that I was visiting with the lads from a prior contract and I moaned that RMS sucked because it could not handle files with hundreds of thousands of records and they laughed at me and said they were handling millions with RMS.

I can actually remember the look on my face, a neat trick when you think on it.

I haul ass back to work and pull out the RMS reference manual and I can tell you that dead trees aside there is one good thing about paper documentation: right above the entry for NULL_VALUES close enough to catch my eyes was the entry for NULL_KEYS.

Yep. You need to specify both. Paul had specifed NULL_VALUE=SPACES. He had not specified NULL_KEYS=YES. The default for NULL_KEYS? Guess.

I kinda wretch inside even now thinking about the astonishing amount of money, work, debugging, and delayed batches that followed from one simple failure to understand one broken thing.

The metalesson shared with "Solve the First Problem"? In programming, never deal with the unknown. This game is hard enough.

Epilogue

The punchline is that I never solved the first failure from Scene I of this tragedy. As my father used to say, "Do as I say, not as I do." We did have a deadline, and I did narrow down the location of the problem in a way that reassured me somewhat that it would not jump up to bite the new code in the rear end. And even in its breech the law is confirmed: we do need to address the underlying problem which I have some confidence I now understand because it still presents problems for the software but it will go away only when bigger problems are solved and they are much bigger so I am keeping my sights set on them.

The Road to qooxdoo Part III: Why It Rocks

2008-12-29T02:23:00.009-05:00

In the episodes one and two we learned all about how a Lisp God and Interwebby know-nothing found a way to support a client needing an enterprise Ajax Web application, said way turning out to be qooxdoo after solid efforts to make jQuery, Dojo, and YUI fill the same bill. But we did not learn very much detail about qooxdoo, we just learned that it comes with lots of doc, lots of examples, a solid community, and that the engineering is first rate. I may not have mentioned this, but it also was dramatically faster/smoother than the others on my acid test: scrolling a datagrid.

Okay, that is a lot as a general recommendation. I mean we did not learn much detail. Until now. Here are four Big Things that really stand out:

1. HTML? CSS? Never heard of 'em.
First comes a shocker. The qooxdoo user codes no HTML or CSS and needs know nothing about those two disappointments. Just as my Lisp compiler reads my Lisp code and spits out optimized native machine code, qooxdoo takes your qooxdoo/Javascript and produces the HTML and CSS to drive the browser. Upside: qooxdoo developers write less code and never have to worry about browser variability again.

To a Web noob like me there is a second big win here: I do not have to learn HTML, CSS, or the vagaries of the different Web browsers my client might want supported. I am a bit of a rarity in my ignorance of those things but soon I should have a lot of company now that qooxdoo has set the bar as high as it has on insulating the web app developer from lower level technology.

By the way, in case you are wondering: yes, there is a widget (two, actually) that will accept classic HTML.

2. I do declare. Not!
The second thing that hits us is a consequence of the first: we are going to be writing a lot more Javascript than we would be with libraries such as Dojo where authors still write HTML/CSS. With declarative tools we sit in Lisp or Ruby or PHP on the server generating HTML. Not qooxdoo (but see next) and not YUI.

I am not at all happy about this but I will live and if it ever bothers me enough I will spend a weekend exploring the qooxdoo markup contrib called QxTransformer. Right now I need to catch up on a backlog of functionality the client has had to do without because of the six weeks it took me to identify qooxdoo as The One.

3. OO is not the Grail, but OO rocks
Stendhal said it best in "On Love" in the chapter on Infatuation: (paraphrasing poorly) we do the object of our desire two injustices, first setting them on a pedestal unreasonably high and then, when they inevitably disappoint, setting them too low. OO is not the Grail, it is just a profound advance in serious application development. All by way of introduction of another stunner: qooxdoo's slightly more substantial object model layered atop the core Javascript OO capabilities.

To be honest this struck me as a bit of a yawn until I decided to start carving up my existing code into distinct qooxdoo classes. This is a very big application and I needed code re-use almost immediately and doing it in HTML with Javascript glue was...well, before qooxdoo I was pretty much starting to build my own abstraction layer atop the core tools--there was no way around it.

But now that I have used qooxdoo's OO to partition my rapidly growing code base I have more than once had that wonderful sense of getting something for nothing after doing a major refactoring (you know, taking all the pieces and throwing them up in the air) and having it Just Work after two or three tweaks. Furthermore, I will not bore you with the details but I have not even started leveraging the full capabilities of qooxdoo's object layer: even more benefit will be reaped moving forward because of some nifty reactive capabilities.

This came as a pleasant surprise. I do more OO in more ways than you can imagine in part because I normally program with Lisp's CLOS which is to normal OO what normal OO is to binary arithmetic but by the time I stumbled onto qooxdoo I had been doing Javascript, HTML, and CSS for a couple of months under a lot of pressure so I never had the time to build a sensible OO abstraction atop Javascripts primitive little OO pretensions. Thus Javascript had endowed me with a bit of child mind and I was able to experience the mundane code structuring win of OO as if for the first time. It rocks.

3. Layout the Wazoo
I may have lied. This may have been the first stunner I felt in my gut. The others I knew would be huge if they panned out, but this was the first one I saw in action working as beautifully as had the other frameworks left me hanging. With other frameworks I could find no way to get built-in layout schemes to use the full window real estate available. That cuts two ways: I either ended up with unused whitespace (and with our app we need all we can get) or my application panels grew so large that the browser itself put up scroll bars, meaning I had to window-scroll my scroll panes around. No, that was not a candidate for production release. Note that to a large degree this is the same as point #1: a framework is not hiding HTML/CSS from me if I am still victim to surprising browser decisions, such as oh gosh that is big let's use a scroll bar here.

How good is qooxdoo at layout? When I popped open the Firebug debugger in FireFox, where by default it begins by seizing application window real estate, the qooxdoo layout engine handled the downsizing impeccably, right down to adjusting scroll bars to accurately reflect the new dimensions.

4. Message In A Bottle

This feature is huge in a small way, by which I mean I could probably roll my own version of qooxdoo's Message mechanism in a an hour but--wait for it--I did not have to. They did. That is huge, the huge thing being that the qooxdoo team really has set out to create a compleat Web application development environment and with such a beast comes pleasant surprises such as their Message mechanism which allows a nice decoupling of widgets: one widget can dispatch "Make it so" and anyone responsible for "it" knows to roll up their sleeves simply by having subscribed to "Make it so".

Beyond the Message mechanism we have the Event mechanism. The documentation of every widget lists the events it might generate and what data the event will carry, and in toto we have an engine with an event model dozens of times richer than pure DOM.

5. I know, I said "4"

Number five is an umbrella win, too detailed to list: as I whined above, this Lisp God is now writing an awful lot of Javascript leveraging more and more qooxdoo all the time. Again and again your highly critical correspondent is finding nothing but good news in qooxdoo.

Yes, I could make a bug report every other day if I had the time, but a long time ago I figured out that I am not looking for a perfect library, I am looking for a powerful library that is fundamentally sound (so anything broken can be readily fixed) and actively maintained (so things actually get fixed).

Summary

qooxdoo: fast, powerful, complete, rich, well-engineered, and it hides browsers, HTML, and CSS. Great doc, loads of examples, fine community. It's all good.

The Road to qooxdoo - Part II

2008-12-13T23:24:00.006-05:00

Last time, we learned how one self-appointed Lisp God with no knowledge of web programming ended up programming the web and then soon enough embarked on a six week grand tour of Javascript frameworks in search of a platform suitable for an enterprise web application and where his tour ended: qooxdoo. That episode ended with a promise of more information on qooxdoo, but first a note on the methodology: there was none.

My first experiment with web programming had thanks to the marvels of Lisp in general and AllegroCL's web tools in particular mushroomed into a full-blown web application aided here and there by jQuery widgets before we slowed down enough to think, Damn, we could benefit from a full-blown Web application framework.

So I eyeballed the usual suspects and ran them through googlefight and picked the most promising one and used that to redo what I had accomplished to date. Which was: tab controls, select boxes, data grids as views into server-side data stores, tree views, text input, tool tips, pushbuttons, pop-up windows, and probably one or two things I am forgetting.

How did I assess the libraries? We needed a professional look, good performance, and maximum productivity. Sorry if that seems blindingly obvious. The productivity assessment has some beef: I looked for lots of documentation and tons of examples describing a rich set of widgets at once high-level, feature-rich, and yet highly authorable which may not be a word but it means I get to override stuff without resorting to back doors.

I considered also two metaqualities: on-line support and internals code quality. Having the Lisp HyperSpec an F1 away is just a little faster than asking for the same information on comp.lang.lisp. ie, Community matters. As for code quality, the good thing about open source is not that it is free, it is that I can use the source to understand things at 3am when the community might be quiet; I can add print statements to debug my code; and maybe I can find and fix a bug in the library. But if the code is of low quality good luck with anything more than the print statement insertion.

It was a rough six weeks. Three times in six weeks I logged onto support groups and said, "Hi, this is my first day..." as I rebuilt a relatively feature-rich web app three times while staggering up to speed three times on three environments. All while the client is getting absolutely no new functionality. I feel bad all the while because they need this stuff now. I am letting them down and there is the strong possibility they will be letting me go: I had been brought on to do the Web front-end. I like learning, but not so much with a gun to my head and a time bomb ticking.

Are we having fun yet? Well, yes. The last framework we tried was qooxdoo and it has worked out great. I can now give the client pretty much what they want fast and looking good once they pony up for a graphic artist because I just saw this impressive example of what can be done with their "appearance" framework. But I ain't no designer.

Funny thing, though: qooxdoo like YUI and Dojo came up short during the evaluation. It had no example of the widget I needed most, a fairly standard widget at that: a datagrid backed by a remote store. Bad sign. Second, qooxdoo did no better on googlefight than does Lisp, and that is pretty bad. Finally, qooxdoo like YUI lacks a declarative programming model [though the declarative QxTransformer has been revived since my original investigation] which means I cannot just sit in my Lisp IDE catching XHRs and tossing back HTML/CSS. This would put a non-trivial dent in my productivity.

So how did qooxdoo prevail? What can I tell you? Bullet charts are useless, expert systems work great as long as expert humans interpret their output, and resumes tell us nothing about the work we will get out of people.

qooxdoo did have a contributed example of a remote data store which pretty much worked and when I looked at the qooxdoo internals to learn how to make it work better I found very good code and rather quickly had the datagrid I needed. Otherwise qooxdoo did have loads of examples and documentation and decent community (I say "decent", you'll say "great"--my standard is comp.lang.lisp) and then we get to the bottom line.

While resurrecting my app for the third time with qooxdoo, yes, I worked just as hard as I did using Dojo and YUI. But the effort was different. With qooxdoo I was struggling to learn a very nice framework that needs an index: I was slaving for six hours to find the simple six lines of code I had to write. With YUI I was struggling with a framework that did not work all that well and scared me more and more the more I looked inside it. With Dojo the struggle was with a good framework but the struggle was such that I suspected the struggle would never diminish and probably increase exponentially as the application grew in complexity.

As for the declarative thing, hey, I am a programmer, I can write code. And Javascript has a lot of Lisp soul, I am not stuck with something daft like PHP or Python. Sure, having to spend more time in a JS IDE means a ten percent productivity hit but ten percent we can do. The terror we developers dread is the tool nightmare death by a thousand cuts two steps forward two steps back rope drag death.

I will blog on rope drag sometime soon, but first let's get specific about qooxdoo. In Part III of my Road to qooxdoo.

The Road to qooxdoo

2008-12-12T20:19:00.010-05:00

[The title of this rather boring entry derives from a fun little survey I kicked off years ago.]

OK, why should you listen to a Lisper on Javascript frameworks? Because we use Lisp so obviously we have better instincts when it comes to developing applications. In fact, this road would not have ended so well had not Lars, a fellow Lisper developing a killer new Lisp Web programming framework called Symbolic Web (SW) brushed aside my enthusiasm for Dojo and mentioned he found qooxdoo interesting, although for now SW uses a little jQuery. "A little" because the whole idea of Symbolic Web is to do most of your programming in Lisp which of course is the only way to go but SW is still emerging from the sea and I had a requirement now (well, six weeks ago, but there is major new release of SW just out I need to look into but as this Road will reveal I am not likely to change horses now because of what I found in qooxdoo (not that SW will not eventually catch up)). Gasp.

The second question might be how a Lisp god and Interwebby know-nothing ended up doing heads-down intense Interweb development. Well, it was the Lisp credentials. A former client using mostly Lisp was having great luck with a Ruby/Rails front end but was looking for more speed and hoped having a Lisp image serve the pages directly would make for better response.

What happened next was pretty neat, and again Lisp conquers all. I told the client I would look at what they had and what they needed for a few days to decide if even I could help them, because I can do a lot of things but one thing I cannot do is exaggerate how little I knew about Web programming and even static HTML eight weeks ago.

A glance at the existing screens and Rails code had me thinking, "No way", but I was curious anyway about even getting an application to drive a web site so I fired up a tutorial on Web app development using tools from my Lisp vendor of choice, Franz.

How good are Lisp, Franz, the tutorial, and the WebActions tool? In the three days I was supposed to be deciding whether I could help the client I managed to execute a respectable fraction of the first component they needed. Sorry, no screenshots, NDA and all that.

So now there I am with a contract and a browser toolbar sagging from the weight of bookmarks to sites on HTML, CSS, and for the hell of it jQuery though I was not sure why but everyone was jumping up and down about it. And then I needed it.

Well, not jQuery. I needed a razzle-dazzle spreadsheet-like grid presenting data from an arbitrarily large table of data held on the server. Now at this point I had already rolled my own Ajax paging mechanism, but something superficial but nonetheless important for that this being an important application was holding me back: my HTML looked like crap. Meta-holding me back was my sense from surfing the Web that getting good results out of HTML/CSS was a black art requiring years of practice which certainly leaves me out. Enter FlexiGrid. A URL which as I write responds "Please contact the billing/support department as soon as possible."

Which brings me to why I dumped jQuery. What I found was small, yes, and add-ons, yes, but a hodge-podge of add-ons does not a professional development environment make. By the time I punted I had contribs from three people and while each was lovely alone together they looked like, well, a great decorator on a multiple-personality binge. And beyond appearance there came a surprising result: FlexiGrid did not offer a "row selected" event. Meta-worse was the author's response. "Right, no row-selected event." Ooops. Next up was jqGrid and that was fine but I still had Crazed Decorator Syndrome going and now I am looking for the next widget I need and a light went on: we are building an application here, not a Web page/site. It is wonderful that jQuery is small, but small is exactly the wrong thing for developing a sophisticated application with a coherent look and feel never mind coherent engineering underneath so widgets can play well together. Enter Dojo. But first...

In case you are wondering where the client went, they are encouraging my search for a pre-fab solution. Probably seeing the gorgeous FlexiGrid widget compared to my ugly efforts did not hurt, but the client has good instincts anyway (they use Lisp, right?) so they are leaning towards my original assessment that I am overmatched if I am going to try to turn out a hot application on short experience with HTML and CSS even though at this point I have wondered aloud to them how much HTML/CSS I might have mastered in the same time I was spending beating my head against pre-fab, this being the classic failed deal we make with the devil when we reach for 4GL: yeah, wow, a complete browser screen in five minutes. Gee, could we get the first sub-item appearing on the same row with the item? No. The client wants it. It cannot be done. That was Datatrieve on VAX/VMS twenty-five years ago, but nothing has changed. 4GLs get their apparent power by making decisions for me which works only if I (and my client) are not making any decisions.

But no, the client says "find a framework" and I decide to try again with Dojo. Again?

I had looked at Dojo before. The word on Dojo was incredibly good. Of course that word was from their Web site. When I tried to confirm by eyeballing the doc I could not find it. I mean, there were some great links, but all the pages they led to were blank. Next! But now I was back, encouraged by the news that OpenLaszlo had just placed a bet on Dojo. IBM as well, but OpenLaszlo is more important because I know they are geniuses, they have a dataflow hack like mine.

Oooh-ooh! Another reason Dojo looked good was a benchmark I had found showing the thing was fast. I have no idea if the benchmark is valid, but the pictures are gorgeous so I trust it. So why did I dump Dojo?

Well, OK, I did not really. It came in second. The question is why did I keep looking such that I found the eventual winner. Easy: too frickin hard to get to work. The documentation is awful. They know this and are addressing it but did I mention we need this now? Overall Dojo fit the bill of being a Compleat Solution with solid backing and a great future, but after banging my head against the wall over something (ah, it was the six hours I spent trying to figure out the "build" mechanism for a compressed production release of my JS culminating in watching a blurry video that finally showed how to do it) I went for a better way surf (as in there's gotta be) and saw YUI touted as a serious solution.

A quick glance at the enormous volume of well-organized documentation and exhaustive examples and the client and I agreed at once to give it a try. And I will tell you right now YUI has the best graphic designers, because their stuff looks hot. Dojo is a close second there, too. YUI is a complete solution with a big backer, gorgeous style, and more doc than you can imagine. How did it get dumped?

Stuff was not working well. Even with all the doc and examples it was hard to get working. When I dug into the source to see why things were not working, I did not like what I saw. And the death knell: YUI abandons the declarative model. Why is this so bad? Hellooooo, Lisp? Remember?

Up thru Dojo I was programming in Lisp 90% of the time, doing just enough JS to get information back to the Lisp server application where HTML could be generated and sent back, even that Lispily thanks to HTML generating macros. And YUI was going to make me give that up and be as hard to figure out as Dojo and I suspect worse under the hood. So regretfully I informed the client my battles with Dojo would be resuming.

So how does this road end up at qooxdoo?

The weekend was coming up. Weekends are mine, so I decide to follow-up on Lars's hint about qooxdoo. qooxdoo is the ultimate threat to YUI because qooxdoo also says Just Code Javascript, and if I am forced to do that by YUI I may as well do it in qooxdoo where the code is as good as YUI's is bad. (I looked at about fifty lines of it for five minutes, I should know.) Anyway....

qooxdoo is amazing. They need help on the graphic design (never look directly at their tab control, use a mirror) but other than that the engineering is terrific. [My bad: they simply leave anything more than the utilitarian look up to the developer, witness this before/after case study.] The documentation and examples are also abundant here, and the support is great. They even do dataflow! Yes, I wish I could spend all my time in Lisp sending over HTML, but two things on that.

One, there is a side-project that brings a declarative model back into qooxdoo. QxTransformer. I have no idea where that stands, but I did see a note on-line where a developer spoke of it in the past tense. [Now resurrected.]

Two, hey, I can write some Javascript. I did ten years of C before I did fifteen years of Lisp, Javascript is not a problem. And it even has lambda, something this guy still does not understand. Lisp/markup would be better, but when dealing with mission-critical stuff (the client is doing valuable data crunching but does not win until they make it accessible) any team can roll up its sleeves and crank out the work even if they have to do without the brilliance of Lisp.

This entry is getting long and I am getting tired and my bartenders are starting to worry about me so I will leave a detailed praise singing of qooxdoo to my next entry, but allow me a bit of a rant. What a mission-critical effort cannot stand is fundamentally flawed solutions that seem to offer a fast track but in fact bleed developers to death. This can be jQuery with an incomplete solution forcing me to try to build an application out of mom and pop contribs or YUI with fundamentally bad code or some other framework that might be just plain slow.

As a developer I am not looking not to work, I am looking to have my work yield quality results predictably. I might spend four hours trying to figure out how to do something in qooxdoo but once I do it turns out to be insanely simple and work beautifully. Moral: their documentation needs an index! But I am keeping my big yap shut because documentation is a bitch and they have done a ton of it and I do not document my stuff so...I keep my big yap shut. But the point is that my work now yields results reliably and predictably and this will accelerate as I learn the framework and even learn better how to navigate the documentation.

qooxdoo not only fits the bill of being a complete solution well documented with many examples but it also has high-quality code under the hood, which both gives me confidence that I have found the right tool and not incidentally serves as useful further documentation. More next time.

Why Lisp Packages are so Easy (Hard)

2008-12-07T12:38:00.001-05:00

Slobodan Blazeski wrote:
> May somebody please explain me with simple words why cl packages are
> hard to use, 'couse I really don't get it?
> The best is probably to hear from somebody who just learned lisp or
> somebody who teaches lisp (do such people exist anymore?) and his/her
> students has problems with packages.

...or a good teacher, defined as someone who remembers what it was like before they knew what they were teaching.

The first problem is one of those immune system deals. Packages have a lot of the same proteins as, say, C dot aitch definition files so the recognition systems of programmers coming to Lisp from other languages (you know, everyone) does its damndest to parse them as dot aitches.

In fact, packages have nothing to with anything but symbols, neither the functions nor global values they might name. More expansively: packages are about mapping string names to first class Lisp symbol objects and dividing this up into discrete mappings called packages so the same string "Hi, Mom!" can map to multiple |Hi, Mom!| symbols in multiple packages.

The second problem is that symbols take us straight to black-belt Lisp in which one must understand the Lisp reader and fancy things like read-time and compile-time and all that. Example:

I remember getting messed over because some symbol was mysteriously appearing the base CL-USER package. I could not for the life of me see where it was being introduced. I turned the problem over to a few trees worth of monkeys and one of them came up with this rule:

"Never use a normal symbol like |Hi, Mom!| in a defpackage form coded in a source file with (in-package :cl-user) at the top which would likely be the default even without that."

Instead, use a string like "Hi, Mom!" or an uninterned symbol like #:|Hi, Mom!|.

Strings are dicey because you might want to export 'banana and be silly enough to code "banana". Lisp will take you seriously and create |banana| instead of BANANA, case sensitivity being what it is except when it is not when Lisp folds 'banana to 'BANANA which is what it does unless you are in Modern (case sensitive) mode.

Did I mention "black-belt"? The fundamental idea is simple, but noobs get thrown off mostly by the non-simular simularity with C and then in the debugging stage by case sensitivity seeming to come and go at random and the whole read-time compile-time thang.

hth, kxo

A Beginners Guide to ASDF (Ha!)

2008-12-07T11:17:00.004-05:00

OK, there is at least one thing about which Lispers are not so smug: ASDF (Another System Definition Facility), the de factor standard for the Lisp library equivalent of Makefiles, and my post today was prompted by ANFOBA (Another Noob Effed Over By ASDF) washing ashore on comp.lang.lisp, and how I almost responded there:

Francogrex wrote:

Hi, I am struggling to understand how to work with asdf.

I have been using it for ten years and I am about 50-50 now on getting it to work. I am assured it does a great job in working with a multitude of Lisps/OSes, and I can assure you it is a disaster at working with programmers.

Come back here and post console output freely, you will need help, especially because one of its great triumphs is useless error messages.

Hints:

(1) It will work. You will have to promise it your first-born, but it will work.

(2) No, PB was not joking: you have to separately push locations onto some magical special variable. Leave off a slash (or add a slash, I forget) and ASDF will take your head off, so don't do that.

(3) No, PB was not joking, the syntax is:

(asdf:operate 'asdf:load-op :cl-pdf)

Why is it not (asdf:load-system :cl-pdf)? In my experience, extremely smart programmers think it astoundingly brilliant when "one function does everything!!!!", remembering the ridiculous op-code first parameter left as an exercise.

Since you asked, this is because extremely smart programmers are tone deaf to programmer productivity; they are so smart they do not even notice when they have created something unuseable, they just brain through it.

(4) ASDF has a featurebug: it loads in the wrong order in order to make McCLIM work. Don't ask. If the author of the .ASD file retreated to the desert for forty days to slave over the dependencies this will not be a problem. Otherwise do what I do: take a sledgehammer to the beast:

(asdf:operate 'asdf:load-op :cl-pdf :force t)

That abandons the whole point of having a build system, but it gets ASDF to work. When for the love of god I am just trying to build this frickin thing!!! that is a nice tradeoff of compile-time vs tracking down and short-sheeting the author of ASDF.

(5) On success ASDF emits more messages than an IBM OS/360 core dump. This is so any errors will scroll well out of view and anyway stand out like a healthy thumb from the surrounding...um, other healthy thumbs. Anyway, do what I do: ignore all messages other than a backtrace and Just Try the Hello World(tm).

If you get a backtrace, Just Post It to c.l.lisp.

(6) If hello world fails, f*ck! But at least now you are a Real Lisp Programmer(tm), this is what we live with.

hth, kenny

The Foisting of An Infinite State Machine

2008-11-18T21:08:00.005-05:00

"I will not allow you to foist this convoluted scheme on the bank!", Sam yelled.

This was fun in a lot of ways. First of all come on this is a bank, three or four full floors of programmers all enthralled by nothing more technical than getting through the day unfired and back on the train home to their families and here is this aging cardpunch nutjob throwback not only noticing Someone Else's Code but caring?!

Second, I did not make that up. Sam actually used the word "foist" without trying. I use SAT words all the time but I am usually trying, you can see my eyebrows arch a little as my language organ reaches for the thesaurus. Nothing shabby about "convoluted", but "to foist"? And Sam tossed that off in earnest in anger in the heat of diatribe sans affectation and it is not clear to me I have ever before or since heard the word used in vitro. Which brings me to the third fun bit.

Sam was Jewish. Classic Orthodox home-by-sundown-on-Friday Brooklyn Jewish and of course he would be good with words. Right, get all PC on me and denounce me but what can you say?, he used "foist" in a sentence on the fly and I think I win on this. So there he is with this classic Billy Crystal Princess Bride accent yelling at me for foisting convoluted code on the bank.

"I will not allow this!" he cried. "I will stop you!"

Oh, please, tho Sam was a sweetheart. Yeah, "was". I heard he passed, Lord keep him, a beautiful soul. I could have choked him in that moment but at least he was attacking me for Code Quality. What was my sin?

I had been tossed a throw-away job of persuading two applications to exchange information reliably when either app was free to collapse in a heap at any time and I was not to lose any information. Nowadays we just pull packages off the shelf for that but this was a while ago. Oddly enough in my first IT job ever I had had to do the same and it was fun because there was nothing theoretical about it, the two applications did even during development reliably disappear left and right allllll the time. Decnet, PDP-11, RSTS/E, 1981... you understand.

What was different was that somewhere in the intervening ten years I had gotten this crazy idea of creating a domain-specific language to make it easier to develop a software Algebra tutor so I had gotten a book on compiler design because I did not realize I could Just Use Lisp. I do not like reading so I did not get very far but fortunately one of the earliest chapters was on parsing so I ended up learning about finite state machines. Oh, sorry.

This post came up because I am doing some Web2.0 programming these days from Lisp and I wanted to send over Lisp sexprs and since Javascript for some reason lacks a Lisp reader I was going to have to parse it myself and after ten minutes of trying to do it without a finite state machine I realize, damn, I should code up a finite state machine.

I will not in this space try to explain what is a finte state machine but it is really simple and powerful and no matter what level programmer you are you need to have it in your, well, toolbox. I think the example I learned on was a pocket calculater when folks still had those. Your initial state is "initial". Duh. Then what happens? Oh, I got "/". Bzzt! Or just ignore it and my next state is still "initial". Now I get a "1". Great! Save that, my new state is...I dunno..."got a digit". We're on a roll. But wait, what other inputs could I get? A "+". Ummm...well, it seems no different, but we'll say "got a plus sign", and indeed when we are done one of the cool things that can happen is that as we fill in the table we may well discover that the row for the state (did I mention that rows were for states and columns were for inputs? Sorry) ..the row for the state "got a plus sign" might be no different from the row "initial" and we first go "whoa, enlightenment" and then collapse the two rows into one. I digress.

But the idea is that we just break it down into cases. A fun example is that we now know whether a "+" is addition or a plus sign (or an error) without really trying, because we have aggrssively exploded the analysis into all these different states like "just got a digit" or "just got a left parens" or...yeah, it goes on, but guess what? Here comes the name! "Finite State Machine"! There are only so many!!! You may think there are a kabillion but after ten you are done, maybe after twenty. And you can collapse inputs (1-9 become "positive digit") and so you have a finite number there. Do you now have 150 cases to handle? Yeah, and each one you can code in your sleep. The alternative is one hyperglutionous mass of conditionals, tests, and branches that will fail on the one millionth execution because of a first time path execution.

Back to Sam.

I have realized that the reliable exchange between two autonomous systems permitted to fail at any point will best be solved using my old friend the finite state machine and I have enthusiastically shared this with Sam. Somehow I am now being denounced for using an algorithm learned from the top book in compiler design. Specifically, I am being denounced for foisting. I will be stopped. The bank will be saved from my convoluted scheme.

Ooooookayyyyyyy.

Well, what can one anachronistic lost in the Torah yamaha wearing card punching goto coding nut job do? Would you believe he could force a code review? Nay, a defense. Sam forced a frickin defense of about thirty lines of the best simplest most powerful correct code that bank had ever seen. Mind you Sam is also the man who produced my favorite line of code ever, using the Vax Basic statement modifier mechanism:

return if flag

That's right, an action at a distance code teleportation straight out of the middle of seventy line for loop using a flag he had named.... flag. You can imagine my reaction. And this guy is challenging my code?

If you are not yet suitably horrified, understand that novice programmers were regularly crashing production runs because no one anywhere was looking at their code and they were going to call a meeting of like twelve programmers to review my foistation of convolution upon the bank? Sorry, my ire is showing. Chase cut to.

Code review/defense: twenty minutes, here it is, any fucking questions?

Yes that was pretty much the tone of my presentation -- I like to consider people skills one of my strengths. And then Sam spoke and you will see why I loved him. At pain of repetition, probably no one else on those four floors would have even listened to me as I bragged on my FSM, but go to the mattresses to stop what he thought was bad code? One in a million. And one in ten million in my experience would have said what he said.

"OK," Sam he said. "That looks fine."

Of course now I am pissed off because I am just getting my turbines spun up for a full bore antler to antler knock down drag out kickin and a gougin in the mud and the blood and the beer round fifteen thrilla in manila finish and Sam is saying...fine?

"That is simple," Sam continued, gesturing towards my whiteboard diagram. "This infinite state machine idea is complicated, but the way it works is simple."

The beauty of that being......no. I would only ruin it.

RIP, Sam. RIP.

URBBR #2: How Bad Is This Book?

2008-09-09T14:17:00.006-04:00

[The following has been... well, only the names have been changed to protect the endangered toads.]

We have a One-Paragraph Test for fiction. Not a complicated test, it involves reading the first paragraph and then deciding. If We are really undecided We can read a second paragraph, but that is a Bad Sign.

For non-fiction We look at the blurbs the author managed to suck out of unwilling blurbers. Well, OK, We do not really want the blurbs to be from blurbers, We want them to be reviews and We know they have cherry-picked the review sentences so everything gets discounted about an order of magnitude.

And there are two such reductions applied, the second for the source of the review. Kirkus Review of Books... well, just quoting them gets a book back onto the shelf. In the wrong section, upside-down, binder in.

Here is the Gold Standard for non-fiction back covers (hmm, the URL goes to the whole book, be clever and click "Back Cover"): The Glory of Their Times. Yes, you need to read this book, and no I do not give a rat's ass if you like baseball just read it.

We turn now to a book I have been asked to review, which will go unmentioned for the same reason I restacked it binder-in:

"This work strikes a balance between the pure functional aspects of Blub and the object-oriented and imperative features that make it so useful in practice, enable .NET integration, and make large-scale data processing possible."

The clever unwilling blurber always manages to say something nice by talking about something else, in this case the language .... instead of the book for 90% of the blurb. What does he say about the book?

It strikes a balance. Wow. How big a put-down is that? Note that the reviewer did not say "a perfect balance" or even "a good balance". And what the Hell does balance buy me anyway?

The good news is that We cannot discount that 90%, Our micrometer does not go that low. But We better check the author of the blurb.

—John Doe, PhD, Researcher, Blubber Ltd.

Oh. The people selling Blub.

The neat thing is that We now know the book is awful and that John was slowest to hide under his desk when the marketing people came around looking for a recommendation.

Unread Book Review: Zen Style Programming, Nils Holm

2008-09-09T06:19:00.006-04:00

On comp.lang.lisp, Nils M Holm wrote:

> I am happy to announce that free pre-release copies of my latest book
>
> "zen style programming"
>
> are now available.
>
> Blurb
>
> The primary purpose of programs is to be understood by fellow human
> beings,

Nonsense! Nils has been reading Knuth or something. The purpose of programs is to model other processes so the most important virtue of a language is the ease of building programs with it. Yes, those programs should tend to be clear but that just requires powerful constructs, long names, and support for the functional paradigm.

Besides, "revise" means "rewrite from scratch anyway because the way this was done is all cocked up" so we do not generally have to understand programs to work on them.

> hence programming languages should have certain properties:

Bzzt! The source of obfuscation is the programmer, not the programming language.

> * Small size and uniformity

Hand-waving! Nils has been reading Graham or something. Small size as a requirement does not follow even if one were to exalt human readability to Prime Directiveness. You could just have good names for functions, like pathname-sans-file. And an editor that knows how to get to a hyperspec.

   * Unambiguousness
   * A high degree of abstraction
   * Achitecture neutrality

The first part of this book introduces the concept of functional programming and describes a purely symbolic language that fulfills these requirements. This language is a minimalisti variant of Scheme whose only data types are symbols and ordered pairs. Nothing else is required to describe algorithms for solving a variety of different problems.

The second part of the book shows how to apply the techniques of  symbolic programming to some problems of varying complexity. Topics discussed in this part range from simple functions for sorting or permuting lists to ``lazy'' data structures, regular expression matching, formal language translation, and declarative programming.

The third part, finally, shows how to implement the abstraction layer that is necessary for solving problems in an abstract way on a concrete computer. It reproduces the complete and heavily annotated source code for an interpreter of symbolic LISP and provides an example of clear and readable C code.

This book contains the full source code to a source-to-source compiler, a meta-circular interpreter, a logic programming system, and to the language that is used to implement all of these.

Damn, that was a lot of work. I should be nicer to Nils. Here's the book, anyway. Let us see what it says:

"The first points are no-brainers."

Come on, Nils, we talked about the hand-waving.

"If a language is too complex..."

Well, by the meaning of the word "too" I think you are on safe ground for the rest of that sentence, but it begs the question of how too is too? Nils needs to make his case or judge someone else's case, not both.

"...programmers will have to look up things in the manual perpetually instead of concentrating on the actual problem."

So now we need programs to be understandable by people who do not know the programming language? I think that was a design goal for COBOL. Anyway, that does not give you a language with a small instruction set, it gives you a language with long, well-chosen names. Like COBOL.

As for looking things up, I want to meet the guy who looks up delete-duplicates when the status bar is saying:

delete-duplicates sequence &key from-end test test-not start end key

Let's see if there are any more self-serving rationalizations....

"The class of problems that can be solved by programs is much smaller. It typically involves very clearly defined tasks like
-- find permutations of a set;
-- find factors of an integer;
-- represent an infinite sequence;
-- translate formal language A to language B;
-- find a pattern in a sequence of characters;
-- solve a system of assertions.
Of course these tasks have to be defined"

Nils must be a professor of computer science. Programs exist to model real-world entities of arbitrary complexity, tho good luck with the human mind. OTOH, simulations of car crashes are admitted into evidence by finicky courts and some people even model human private Algebra tutors. Unfortunately those things are a ton of work which is why you will not see academics doing that.

Now about the title. Zen zen zen, what an affectation! Zen this, zen that. I am going to write a book on The Zen of Ordering Take-Out Pizza.

What is zen about zenlisp? Never justified, never even addressed. Does the author think zen means small because it is a small word? I like that, actually. I tried googling for synonyms: oops. I think they would be pleased down at the monastery.

So much for the name. How about the language itself? Raison d'etre?

Zenlisp is similar to Scheme, but simpler.

That sounds like being rescued from a desert island and then dropped off in the middle of the Sahara. No, the only reason ZenLisp exists is because Lisp is a venus flytrap of a blackhole delivered by John McCarthy an alien from outer space sent to stop good programmers from Actually Programming. Apparently Grace Hopper was just a few lines of code away from anti-gravity and time travel so They sent John down with Lisp so any good programmer would get so excited about the language itself they would stop programming time-travel applications, take out their CodeWarrior CDs, and start implementing a Lisp using (wait for it) C.

The ones who really want to avoid programming will then write a book about their language.

So I wonder what Nils will do now.

AA, BB, CC, and DD

2008-07-25T03:00:00.004-04:00

I am not making it up. Those were the datanames in the code I had inherited from the Wunderkind. So this will be the most boring stating the obvious war story I ever write but that is not why I am writing it. I am writing it because the Lisp subset of the human race is at it again running factual red lights with their prior conviction feet to the pedal.

Yes, kiddies, we revisit today The Unbearable Impenetrability of the Lisper. The cool thing being that Arc is again involved albeit peripherally in this latest train wreck of human comprehension.

Back then the Maddening Crowd was utterly fascinated that I liked Arc which I did not but they were not to be denied, the best part being those people who responded to my objection to the crowd's misperception by saying yeah I saw you were flabbergasted by everyone thinking you liked Arc so tell me, you like Arc?

No I am not making that up. Twice.

This time it was my scrimshaw-ready datanames, sample below. Some months after my exploration of Arc I had a laugh as I whipped up a DSL for my Algebra software and found myself approaching Arcitude in the brevity of my names. I posted something to comp.lang.lisp inadvertently loosing The Hounds of Lisp Density. A sample:

(hard

(dsb (b x) (rp 2 (rv))

(m/ (m^ b (m* (r2 7) x))(m^ (xqv b) (xqv x))))

(dsb (b x) (rp 2 (rv))

(m* (m^ b (ms* (r+ 7) x))(m^ (xqv b) (ms* (r+ 3) (xqv x)))))

(dsb (b x) (rp 2 (rv))

(m*eo (ms^ b (r+ 7))

(m^ (xqv b) (ms* (r+ 3) (xqv x)))))

(dsb (b x) (rp 2 (rv))

(m/eo (ms^ b (r+ 7))

(m^ (xqv b) (ms* (r+ 3) (xqv x)))))

(dsb (n d) (rv 2)

(w (k (r+ 12))

(m/ (m^ k n) (m^ k d))))

dsb is short for the Lisp destructuring-bind.
m/ is short for make-fraction.
m*eo is short for make-product reordering the factors randomly (either-order)
etc etc

I gave a tip of the hat to Arc and explained that I had done this before (in C) and that this code (to generate randomly many varieties of Algebra problems) was a known PITA and even in C I had used the C preprocessor to likewise make the coding manageable.

Enter the Savages of Comp.lang.lisp.To a geek they lectured me on the importance of nice long meaningful names, or as His Sulzberbergerness edified me a ways back what Confucius called The Rectification of Names which is not quite the same thing but I can never resist name-dropping either Jay or Confucius.

I responded to the jackals nipping at my heels that yeah I know but in this case with vast incessant repetition of a small collection of opcodes that recognition would not be an issue and that it was much better to diminish the low information content (cue Shannon) of long names (what exactly does make-product add to m* in a context where a leading m is used only for makers?) and learn a dozen opcodes which were mnemonically and predictably built anyway from atoms such as M and * and EO.

The universal response was that longer names were better. My universal response was that these special circumstances flipped the arrow on that otherwise sage rule.

The universal response was that longer names were better.

Trying again, my universal response was that I agreed, but would anyone like to address the salience of the special circumstances I had suggested were germaine, perhaps explaining how they were not special enough or too special or the wrong damn color?

They all responded that longer names were better and I really started to enjoy things at that point. I was reminded of this desert spider that had a routine for burying any captured wasp and it involved positoning the wasp up just so and then going to dig a hole and then dragging in the dead wasp and these researchers would move the wasp a little while the spider was digging so the spider would be thrown off and start again by repositioning the wasp and no matter how many times they moved the wasp while the spider was digging the spider would just start right over readjusting the wasp and then digging the hole.

The topper was I myself was a staunch proponent of good long names. Tilton's Law of Programming:

Spend more time on the names you choose than on the algorithm.

As for the specific quality of length, we have Tilton's Rule of Abbreviation:

Abbreviate no name less than seven characters long and then only if a good abbreviation is no more than half as long. Rounding down.

Which brings us to AA, BB, CC, and DD. I was working as a body shop consultant in easily my most Tall Building job ever. One day this new guy came on board, totally not GQ, scrawny, smart, energetic. I had no idea he was a first round draft pick, destined for greatness. An employee, by the way. Tom. He dives in and starts churning out a front-end application, learning the HLL and OS and tools all at once, a man after my heart.

At one point he mentions to me a problem. He wants users to be able to make discontinuous menu jumps, say, go sideways without backing up to the menu above (yeah, this was the good old days of modal interfaces) and the programming language would not go sideways. ie, He was using HLL recursion to handle nested menus.

I suggested the obvious: no, you cannot call sideways in a structured language, you have to return from the called function with a "message" always checked by the caller to see if the user should be taken somewhere else. Tom yelled Great! and tore off to code it up.

A couple of weeks later I have inherited the system. Tom was smarter than I: as soon as he had chalked up the win he told his boss to "get some consultant" to maintain it. Moi. So there I am working on the first RFE and I am perusing the code trying to figure out how the hell it works and I find myself slowed a bit by the data names that pretty much controlled everything: AA, BB, CC, and DD. Come on, you think I could make up names that bad?

So over I wander to Tom's desk, clear my throat, Tom looks up.

"Tom," I said. "I was looking at the code. AA? BB? CC? DD?"

Tom burst out laughing.

"Those are just temporary variables!" he protested, laughing even more.

"Un-hunh," I replied. "And they control the entire program flow."

"Well, change them if you like."

As I said, Tom was a smart cookie, he was wiping his hands of the whole deal.

Now it turns out that one of the things I like to do when working on OPC (Other People's Code) is to stare at it and stare at it and when I see some crucial variable playing a big part in things and its name is getting in my way I pick a better name and do a global change, rinse, repeat until the damn code makes sense. The only reason I had gone to talk to Tom was the same reason we pay to see a two-headed sheep at the carnival.

Two hours into the renaming I had come up with decent names for AA, BB, CC, and DD (that last one turned out to be three different variables) and I was making the global changes eyeballing each as I stepped through the source and after one such change forgive me I will never in my life remember the specifics but imagine you have just made the substitution and are now looking at a line of code that says:

total-weight = total-weight + this-length

I did not feel completely comfortable with that edit so I called Tom over. Did I mention he was a smart guy? Two seconds into explaining how I had gotten to that point he yells out, "Great! I gave up on finding that bug!"

Smart guy. Tough bug? He just moved on and chalked up the win. Let some consultant take over the code and run into the bug and think they introduced it.

But there you have it. A bug so hard to find that a very smart programmer gave up on finding it and someone who did not even know the bug existed changed a few datanames and the bug positively jumped off the page. Explaining the corollary to Tilton's Law:

If the names are right the algorithm will write itself.

Now if only some walking fencepost from c.l.l will post a comment saying... well, that would ruin it, would it not?

Now Batting, Dave. Why is Mariano Rivera not worried?

2008-07-18T15:16:00.004-04:00

The venerable Mr. Roberts draws on his profound mastery of things algorithmic as he intones:

I'm actually quite conscious of cons-ing, but I also believe that obsessing about cons-ing leads to premature optimization and far greater numbers of bugs.

We are fortunate to have the benefit of his wisdom and it occurs to me that we can keep talking past each other enjoyably for weeks like this so i will not suggest we try to establish a concrete example where he would code append and I would code nconc and ruin all the fun. But it is a good question especially if we agree (we do) that as a rule the functional paradigm utterly Rocks the Casbah so how on earth am I going to get a list back from some function I call and not be free to nconc it? The moral being I am right (will it never end?): if we try to find a point of Actual Disagreement(tm) we will fail so let us not.

Getting back to the fun abstract theoretical conflict, the problem is a policy of copying which leads straight to the solar power objection aka death by a thousand cuts which in turn leads us back to the simple question of why we do not just code the right operator at every turn? Genius is in the details which means we are always on our game bringing me to Mariano Rivera.

For those who do not know Mariano plays baseball and is starting to be called the best ever at his position, namely "closer" which means we won the first eight innings and do not want to lose the game please do the rest and he does. Even in the seventh game of the World Series that he lost every hitter broke their bat but the three balls involved known in the game rather morbidly as "dying quails" found their way to the grass. Hard to feel bad about shattering three bats. I digress, Mo is good.

Genius. Details. The great Nconc vs Append War, Roberts-Tilton III, nothing like this since Ali-Frazier, Affirmed and Alydar.

It was Just Another Game. The Yanks had won, Mo had saved. They always collar some poor sod and keep him from his shower and car service home to make those of us who pay to watch baseball on cable TV feel better and tonight poor Mariano's number (42!) came up and by the grace of God I pulled my face out of a carton of cold noodles with seame sauce just in time to witness a perfect moment in sports broadcasting and as well a master class in excellence any time any where.

The announcer on the field for the interview went for a home run question, something way more astute than the usual "How did it feel getting the big out?" and he came up with a beaut:

"Billy Bob is a rookie and you have never faced him before, there must not be much of a book on him," the announcer observed. "Was it tougher facing an unknown quantity?".

"No," Mariano replied. "I went out during batting practice to watch him hit."

Mariano's disclosure that as a Hall of Fame shoe-in and multi-millionaire perennial All-Star participant he had looked down at the lineup and seen an unfamiliar name and broken off his clubhouse routine to go watch some ridiculous kid take batting practice had no small effect on our seasoned, professional, expert baseball announcer:

"You did that?!!!!!!". He blurted out, hopping a little in the air, lurching towards Mariano. "You went out to watch him take batting practice?!!"

Now it was Mariano's turn to be amazed, leaning back a little to create a margin of safety.

"Of course," he replied. "I get paid a lot of money."

Doodleoodleoodleoodleoodleoooo. Game over. The announcer (and I, I confess) had been seriously schooled.

We sit at home and watch these athletes precisely because what they do is so wonderful to us and we endow them in our minds with magical and wonderful qualities because at an early age we were robbed of angels and dragons and Santa Claus and even monsters in the closet and trolls beneath the bridge and then every once in a while we learn that the miracle strikeout we saw with the bases loaded happened because a multi-millionaire punched the time clock, checked his glove for broken laces, sharpened his spikes, and checked the opposing team's roster for any names he did not know.

I say learn the damn language.

Fear No Evil II, or Why Solar Power Sucks

2008-07-18T02:55:00.004-04:00

Just a quick note cuz I remembered something that answers Dave's well-reasoned comment:

My own personal rule on this is to use functional programming whenever possible and let the GC deal with it.

Not much of an excerpt, but it says it all: modern Lisp GCs truly Rock the Casbah, and I myself am on record as denouncing the flipside of this blogcoin, consophobia. Actually, there may be a problem, Dave says "whenever" possible and perhaps then we agree!

But "whenever possible" has a ring to it like "oh, you know, all the time" and I actually had a good reason for not doing that, I just forgot it. Has to do with solar power.

The reason solar power sucks even though there is an overwhelming amount of it is that it is diffuse. Yes, enough sunlight falls on Kansas in a month (I am making this up) to power Earth for a year, but just try to find a solar panel as big as Kansas, never mind deal with the Kansans or the inefficiency of the transmission to Australia. I digress.

The problem with a "Damn the consing! Full molasses ahead!" policy is that when ones application drags to a halt there will not be a bottleneck to fix. There will just be hundreds of places one was too lazy to think about whether one could use the right construct. And correcting them all will be as painful, tedious, dreary, mind-numbing (sound like fun yet?) and thus as bug prone as it would be easy and fun whenever reaching for a list manipulation tool to challenge oneself with, gee, can I cut in the afterburner? Stand back, GC! Captain Destructo is in da house!

That is a practical objection. My ethical objection (viz, hey, learn the language) is not all that far behind: jeez, I am building this list right here in front of my eyes, why am I not using a destructive operation for the next step? Do these same folks all have a copy-n-sort function in their personal toolkits that begins with a copy-list on the sequence to be sorted?

No, they do not. Why not? Hello.

My apps periodically get bogged down. I wait until they do and then dive in for what I like to call Speed Week after a regular TV extravaganza on vehicular racing. Normally this turns up interesting algorithmic gaffes I do not feel too bad about but one time the smoking gun was the most innocuous little needlessly copying statement you can imagine write at the center of the heaviest hitter bit of code you can imagine. This is pretty much the opposite of the solar power argument -- there was just one little guy to track down -- but without the code profiler offered by AllegroC I am not sure how long it would have taken to find that.

And now I come back to the main point: er, exactly how hard is it to know when one can use destructive list operations? So why not use them? Only one excuse comes to mind (fear) but always copying is not even a clear win (ie, it can totally suck at times) so.... learn the language?

If I was going to cut the Copying Cowards some slack I would say, shucks, not everybody writes insanely intense tight loops. But then I remember a point I made in FNO/1: we get noobs all the time struggling with slow code because of excess copying. Perhaps complex applications and tight loops are orthogonal?

I say learn the damn language.

Chomsky! Pinker! Get In Here!

2008-07-14T02:25:00.005-04:00

[OK, OK, it is a Lisp blog but if I can refute Chomsky's entire body of work on language acquisition in one post, why not?]

These academic PhD geniuses have a problem. They are geniuses. They think they can figure everything out by thinking, like Feynman fixing the neighbor's radio at the age of three or something close to that by thinking. Great story, buy the book. Forget which.

Chomsky and Pinker merely repeat the error of the AI crowd who sniffed at NI (I just made that up, it means "natural intelligence") and said, "NI?! NI?! We don't need no stinkin NI!". They said that because they were geniuses they knew that what NI did with pretty much organic chemistry they could stomp with (drum roll, please) reason. Ooooohhhh, excuuuuse me!

I guess I can leave Pinker out of this, he did not add anything to Chomsky, he got drugged in only because a correspondent mentioned his "work" on blank slates. I love that "work": Oh, let me lean back and draw on my pipe and I have a PhD so if I think babies learn something too fast they learn them too fast, dammit! As if. And believe it or not I am coming to my point. But first a science trick, a prelude to my witnessing of the refutation of Chomsky. It has no bearing on anything else but it involves a cute girl and something I could not believe when I saw.

There I am in a sports bar/shore bar celebrating a great week almost delivering my world-changing Algebra software when I see a lovely lass drop her drink. Three or four feet, to the floor. It was a plastic cup, which helped a lot come to think of it, because a glass glass being more rigid would not have absorbed the energy and there would have been a wicked rebound preventing what happened from happening: the drink landed upside down and just stood there, trapping half the drink inside. The bad news being there was no elegant way of finishing said drink, but that is not my point. Did I have a point?

Well, if I had to have a point, it would be that no one else was all that impressed. Clearly I need to spend more time in bars late at night watching drinks being dropped and I do not think it is possible to spend more time in bars late at night than do I so I guess I need to find bars with clumsier people, good luck on that.

Oh, and the other useless and irrelevant to the demolition of Chomsky/Pinker thing I learned is that it takes just one unauthorized speech on world affairs (well, he was mumbling pretty badly, I am not sure it was on world affairs) over the unattended band microphone to get escorted unceremoniously out the front door at the sports bar in question. Right, Chomsky.

So I look up and I see a skateboard competition, lucky enough to be watching the winner's winning run, and lucky enough for my brain to clear long enough to understand what I am seeing, a Johnsonian kicking of the rock I refute you thus Dr Berkeley of Chomsky/Pinker by some sixteen year-old if that. What did he do, and why did my brain have to clear? Two good questions.

What he did was give himself a push across a flat floor in a reasonably small room (no gigantic ramps and concomitant acceleration/stored momentum) and upon reaching a ramp at the same time push off in such a way that the skateboard (unattached to his feet for those who do not know) elevated with him to a height just above a rail (er, just an elevated horizontal pipe mebbe a foot off the floor) such that he could land on the skateboard in midair just as the tip of the skateboard landed on the rail and using his/their forward momentum proceed to slide along the rail for a couple of yards before again levitating off to land somewhere else to commence another trick.

And before the anti-blank-slate morons can scream "Oh that is easy, I have the algorithm right here" allow me to break their hearts and add that as he did the initial elevation springing the board into the air with him (a yawner in these competitons) he took the trouble to pop his deck into a barrel roll (or two, who knows at that speed?) before regaining it to land the rail toe-slide. If the MIT geniuses think that is easy lemme just say that I have spent years in Central Park watching the same people practice on skateboards for years and they still cannot jump over a small backback and land on the skateboard on the other side.

So... these people have skateboard organs? Or do you think learning to barrel-roll a jump into a rail toe-slide is more important to them than is mastering language to an infant so their motivation is stronger? This would be a reasonable belief if one has never been in the presence of an infant who will join in any conversation it hears when it neither understands nor can generate one intelligible syllable.

Oh, and why did my brain have to clear? Because what I saw was a commonplace: inconceivable performance easily explained by sufficient practice. If a recent blog had not been on my mind I would not have given the skateboard performance a second thought, nor the motorcycle-flipping organ manifested in the next story on ESPN.

Come to think of it, this is about why Lisp wins: we simply are not as smart as we think we are. I like to say that I cannot write good code but I do know when I have written crappy code. Recognizing is easier than generating. Hell, crappy is just one bit, right? But how do we produce a huge stream of uncrappy bits? Good luck.

What we want then is an agile language that lets us create awful code really fast and fix it as fast as we identify the problems. Pretty soon our software organs cut in and we are creating not-so-awful code up front. Even better.

Now I just have to figure out an elegant way to finish a drink upside down on a barroom floor.

Buzz off, Charlotte

2008-07-09T02:39:00.003-04:00

Omigod. I just made up that title because of the subject of this post and not five minutes ago I ended up daydreaming about a girlfriend I had back in college named Charlotte and as you will see there is no connection. Scary.

Anyway, it is 2:38 am and I am in the middle of a trivial but unwelcome re-de-refactoring with a scant few days to go before a self-imposed but jeez I gotta get this frickin thing out the door it is the middle of July deadline and it is an educational app and how far is it from the middle of July to frickin September?!!!!!!! I digress.

It is 2:38 am and a spider I swear to god no more than 2mm across is dangling in front of my flat planel trying to figure out this code (the last bit of the last line, to be precise):

(defun alert (s oks)

(tk-format-now (conc$ "tk_dialog .oops {} {"

(format nil "I thought you were solving for \"~c\"? I do not see it."

(key-char sfvar))

(format nil "} {} {~s} {~s}" oks oks))))

Charlotte (look it up furriners, E B White) is freaking out over the double occurrence of the variable OKS. I know because the little hussy has lowered herself I swear to God ten feet from the A-frame ceiling of my ranch rental all the way down to within a foot of my desktop and stopped. How still is the air in here? She was swaying no more than another 2mm. I digress. But how does a 2mm spider pack ten feet of gossamer into its, well, gossamer pack?! I don't care how thin it is, it is stll ten feet!

Charlotte was both deeply concerned and a sufferer of tunnel vision. She ascended to the top of the flat panel (I have it angled down to my slouched position a few inches above desktop), gained a foothold or five, and then inched...no, mmed over and redescended to get a better look at the first occurrence of OKS, just to be sure. It is possible she then pirouetted on her thread to ask why I did not check the CLHS on FORMAT to find out how to back up one argument and re-use it, but I cannot be sure. If my eyes were better, I could tell you.

Well, I am in a hurry dammit and you know what I think I have the strength to type oks a second time and no I do not think the redundancy will lead to the loss of a satellite but if a damn spider complains I know enough to listen... ah, there it is ~:*, back up one arg.

Now buzz off, Ch... hey, where'd she go?

It's the shoes, Jeff! It's the shoes!

2008-07-03T02:07:00.004-04:00

My buddy Jeff wondered aloud on comp.lang.lisp why We Lisp Gods were so much better than the rest of you at adapting to alien environments, aka other languages. Without coming down in favor of any, he offered several possible explanations:

We are just smarter than you un-Lisping dopes
We are accustomed to learning other languages because the food is OK in soup kitchens but the decor? Omigod!
We are older and you can't teach an old dog new... hang on.

Based on recent copious experience, you can safely forget #1. Lispers are precisely as dense as other programmers, and if anything more so: as the title suggests, we are being carried by the shoes. Anyone both smart and using Lisp is named Graham or Morris and either has or split over forty big ones, and no, a thousand smackolas is not "big".

#2 is a fine answer but is boring and knocks the legs out the upcoming argument so let us declare it false. That was easy

#3 is part #2 and part what I joked: old may be wise but it sure ain't adaptable, it already knows The Right Way, it aint listenin.

That leaves the title of this entry, a reference to one (or more?) humorous Spike Lee ads for Nike sneakers in which he insists the sneakers explain Michael Jordan's Unbearable Lightness, to which we see his Airness reacting with feet-on-ground dubiosity. IIRC, but I think I do.

But in this case, I think for once Chomsky might find a home. Chomsky decided that because he Chomsky could not understand how infants picked up language that there must be a magical device at play, a language organ in the brain dedicated to language acquisiton just as the spleen is dedicated to damn I wish I had not slept through biology.

Chomsky had a tiny little problem with his nonsense in that when one looks at human languages -- any of which can be picked up by any infant raised in its midst, whatever that infant's genetic tree -- well, the structural variation is infinite. I remember one universality they found: adjectives tend to appear very close to the nouns modified. Well, shazaam!

So the Chomsker has a problem: this super-duper language organ has to be hardwired not to learn any particular syntax, it has to be hardwired to learn every syntax. Ok, that makes no sense, shall we try again?........... um............. er....................... I got it! It is hard wired to learn an abstract syntax! If only we could find one!!! Chomsky failed except for that bit about modifiers miraculously always appearing close in sentences to the thing being modified. Yeah, how come no language ever came up with "The little giant slowly balloon over drifted the mansion red white."? Because the language organ rejected all those! Where was I?

Ah, flexible language acquisition, computer or otherwise. No, it is not the brain, it is the shoes. Basketball no, programming yes. No, Michael did not reach new heights because his sneakers opened new doors for him, but yes: expose someone to Lisp and (as many before me have observed) they become a better programmer.

Chomsky saw the infinite variability of actual syntaxes as a severe objection to his hypothesis, and he was right to do so. Indeed, look at it from the other direction and we see the light: forget infants' ease of language acquisition, look at the difference between those who know one language and those who know five. Something happens after one has learned a bunch of languages -- acquisition of the next becomes a snap. No, I do not have that backwards. There is a small effect of "hey, anyone who has learned five just digs languages and does because they are easy for them", but I learn natural languages insanely easy and get a huge kick out of stumbling thru a conversation in French or Mandarin and learning a new language is still hard for me because circumstances never led me to learn more than what I needed to get out of high school.

No, what is going on is the same as any learning process: the more we do the more the brain wires up metaskill at that activity. Babies are not born with a language organ, they just happen to have nice blank slates and speech is to them obviously something utterly compelling to master, it seems to control these big creatures moving about before them. Nothing like insane motivation for apt learning, eh Dr. Chomsky? But later on in life when a second language is an amusement? Acquisition is brutal. Where's the language organ now? Withered? Why does it come back after I have learned four the hard way? Ooops.

Was I supposed to be answering Jeff's question? My bad. Chomsky was wrong, McCarthy was right: a language in which code can be data and verse visa is one in which the programmer is stripped of structure and stricture and as Sartre warned us is not free to be not free. In mastering any art true freedom comes only within the constraints of the form, but Lisp offers us no such handholding. Some brilliant wag once observed that Lisp gives us a million ways to do anything but (1- million) of them are wrong -- a working Lisper like one who has learned the hard way five natural languages learns a metalanguage of programming that would delight and disappoint Chomsky, because the learned metalanguage is just one more demonstration of the brain's ability to find patterns in whatever it is immersed in long enough. ie, Organ, schmorgan, neural nets simply rock.

Lispers are not smarter, they just fell into a pool and learned to swim -- ah, good analogy, nothing to hold them up, had to make do in a medium that would not support them, they had to add energy to the system to stay afloat, something like that. Lisp not only supports every paradigm, worse, it even allows any paradigm you can invent one as I thought I did with Cells until I looked up the prior art, but that does not change the fact that while working on a little GUI geometry problem a new paradigm jumped into my code... deal with this kind of language and... well, imagine sneakers that really did have wings. Wear those for a year and then climb back into ordinary sneakers...I'll give you a minute to think. What happens?

Disappointment? Some. Depression? No. End of flying? Never. You have tasted flight, you know what is possible, and you want to be there again. Now the analogy breaks down, because guess what? Lisp is just another language. When you do not have it, you can Greenspun it. When subjected to a toad of a language you grouse for fun but you also have code to write and you shrug off the downgrade and fly with the sneakers you have.

In my first programming job I astounded the team so badly they kinda fired me, gave me only the maximum raise and blew me off when I asked for more because in nine months I had finished every system they had in development. At my going away they said they did not go out on a limb for me to exceed the maximum because they knew I would be leaving anyway. They did not know how much I loved the company bowling league or softball beer league. What was my secret in this Cobol shop? I had been programming in Basic on an Apple II.

One day I came upon a senior guy (OK,they were all senior, I had been there three months at the time) staring at my code as he tried cloning it to his assigned, well, clone. Concerned, I paused and asked if there was a problem. He waved me away, saying, No, I just have never seen Cobol like this. I understood. I knew even as I did it that I had been programming Cobol in Basic. I think I have that backwards. I remember asking them -- I was just learning Cobol at the time -- if a subscripted variable could be a subscript and I remember a very nice twenty-year Cobol veteran asking me very gently if I might not be making things a little too complicated. Nah, just trying to do Basic in these Cobol sneakers.

Lisp then is a substitute in itself for learning five natural languages, because it itself is not Chomsky's language organ, it is the universal syntax Chomsky had to identify for his imagined language organ to have been hardwired to learn. Lisp is the programming language embodiment of Sartre's curse: we are not free to be not free. Program this language and one will first use the wrong language feature at the wrong time (hang out on comp.lang.lisp to witness the hijinx of Lisp noobs trying to do things "The Lisp Way") and eventually get to the point where one can look at problems and see the paradigms they bring with them aided always by your language which is being utterly useless at telling you what paradigm to use, it could care less.

ie, It's the shoes.

30Day Death March: If They Can't Take a Joke, F**k 'Em

2008-05-30T16:52:00.005-04:00

The title? Part one is someone's idea on how to distract programmers from working so as to make it even less likely we will get anything done in thirty days but at least we'll have company. Part two is how we pull it off and is what we used to say about the customer on my summer job in the automobile brake repair shop when we felt our work was a little sketchy.

I poked around on the feed and can confirm what someone said: I totally killed myself over the years by thinking certain edge features were indispensible. In my head an imagined reviewer was going batsh*t over something that could be better, like a Consumer's Report automobile editor knocking off two stars from a sport sedan review because their test football linebackers found access to the rear seat problematic. Did you notice it was a sports sedan?! The linebackers can take the frickin bus, OK?!

Even knowing this I did it again this time around (this is version two of something I did in the last century) and I blame Lisp because it made it so damn easy. At first. I failed to notice quickly enough the ensuing death by a thousand cuts for which I thus had signed up. That sentence reminds me of diagramming sentences in the ninth grade when I should have known I would be a social failure when the English teacher and I were doing high fives over the corner cases while the other kids in the class were silently writing me off as... I digress, just watch out for Lisp, it is a two-edged sword.

The crazy thing is that I also know damn well and will tell anyone who asks that this is one of those 80-20 or 90-10 deals, pick yer own damn ratio, OK?: almost all of the value is in the first feature. [kick kick kick] That was for me, not you. Who are you, anyway?

One caveat: sitting in my cubicle above cloud level one day on one of five floors dedicated to IT drones I was stunned to overhear an intelligent abstract discussion of technology. Someone had a theory. I love a good theory. This one said, Don't ship too soon. The point was and my experience had already confirmed that shipping costs. Period. Suddenly one is working on all sorts of crap that does not move the product forward. Once one ships being live costs. This microISV stuff is about one-person operations so guess who is dealing with the cost/burden of being live? And guess another thing: how completely ineradicable are first impressions? Can I write a survey question guaranteed to elicit unbiased responder input or not?

Oops, what am I? The Anti-30day-Christ? OK, let me finish on a positive note. True story.

"These drawings are wonderful," marvelled aloud the pre-school parent to the children's teacher. "You have a great gift for art instruction."

"No," the teacher demurred. "I just know when to take their drawings away from them."

We few, we happy few Actually Writing Applications have as our mission should we choose to accept to take from ourselves our own applications.

"We Can Live With the Way You Handled That"

2008-03-22T17:45:00.006-04:00

Tilton's Lemma had come under attack and the title of this entry was the happy result. You forgot Tilton's Lemma? Here ya go:

Refinements to requirements cannot vary fundamentally from the original.

The point is that careful design is worth the effort so that software can be successfully rolled out even though users naturally produce RFE after RFE during user acceptance testing. No matter what they come up with the software will adapt to it with a minimum of refurbishing.

So what was the attack? We must begin with the launch meeting of the relevant project, where my conduct was such I rarely again got asked to meetings, but as a tease I will share that this was one of the most profound God is speaking to you experiences I have had in software development, one of those moments where one senses forces unknown at play.

I was being given a nice new standalone task perfect for a cowboy contractor such as myself who was good with code but not with teams. My manager Frank would be perfect for such a loose cannon, one who cared not about how loud I snored when I catnapped or at what hour I showed up or how badly my lunch reeked as I ate it over the keyboard but only if the code worked so he could sleep at night.

Just one problem. I walk into the launch meeting and sit down having no idea what is happening. After thirty minutes of sitting quietly at one corner of the table having nothing to do with any of the discussion I have deduced that a foreign branch of the bank has accounting software that will be shut down and covered by the software at our central location and that I will be writing the interface to take the feed from an off-the-shelf humongoid accounting package and feed it into the central system in such a way that it replicates their existing system.

I have grown increasingly intrigued at this prospect as the meeting drags on because our group has more than a few people assigned to supporting the massive off-the-shelf system (shall we call it Isis?) and it seemed worthy of months of apprenticeship. On top of that I certainly had no knowledge of the foreign branch's system. Best of all I knew less than nothing about accounting. Okay, no, best of all was that no one was talking to me at the meeting so I developed this crazy idea that they were just going to turn to me at the end and say, Need anything else, Ken?

Then Tony the Senior AVP turned to me.

"Need anything else, Ken?"

Left to my own devices for the first 99% of the meeting I had had time not only to deduce its ending but also prepare an answer. My fans from comp.lang.lisp can tell you His Kennyness likes nothing better than a good analogy and it had occurred to Himself that the developing situation resonated strongly with an image that formed spontaneously in his mind of a couple with a dog on a leash standing on some corner in the middle of Manhattan deciding how best to get to the Botanical Gardens in Brooklyn and after considerable discussion and perhaps even a consultation with maps street, bus, and subway settling on a means and path of transveyance and turning to the dog and saying, "Lead on, Fido!"

Hence my prepared answer, before not one but two AVPs and the users on conference call from Canada:

"Woof! Woof!", I offered, and looking back I regret only not having thought to wiggle my ass in my chair to signify enthusiastic tail wagging.

When it became clear that the import of my barking had eluded those present and had at least stunned Canada into silence I expanded on it by observing that I knew neither the sending application nor the application I was meant to replicate nor the entire subject of accounting and concluded by wondering aloud if the guy outside tending the falafel cart might not be a better fit since he at least knew how to make change.

The Canadians had heretofore been utterly thrilled at the idea of having their accounting taken over by our central gang that could not shoot straight (I'll wait while those of you who missed the irony go back for a look) and now they had White Fang as the contractor assigned to the task, confirming their worst nightmares.

They agreed to provide a thorough specification and the AVP and I ever after when meeting in the hallway would bark at each other in greeting and at his going away dinner -- well, after the second round of shots it was pretty much non-stop howling, thank god for private rooms, we should have had one. I digress.

Remember: this is about users not being able to change requirements discontinuously. So now we know how I ended up with the most beautiful requirements from which I have ever worked. Requirements? Bah, this was pseudo code. Just brilliant. They identified a dozen kind of transactions/situations/circumstances I could expect from Isis and they specified exactly how each should be handled.

And this "exactly" was not just exact. There was this tremendous regularity in how they specified the handing of each case. Each was expressed cleanly as so many debit this, credit that, debit this subtransactions. These subtransactions just begged to be viewed as microcode for the larger instructions specifying how each case was to be handled.

So that is the code I wrote, after effusively praising Team Canada to themselves and anyone else I could find. The high-level code read Isis transactions, identified the high-level case, and invoked its handler, each handler dispatching one or more microcode subtask handlers. We are talking easy work and soon we sail into user acceptance done thoroughly for a couple of months because I mentioned they did not trust us. I also know it was a long time because by the time the call came through with the title of this entry I had forgotten about the entire issue to which they referred, which was this:

After a week my bugs had been rooted out and IRs were being handled by letting them know how they had screwed up the test. Frank was happy, I was snoring, life was good, we had time to buzz about this new write once run anywhere language called Java that would let browsers run applets.

Then well into the test when the IRs had slowed to a drip I got a call from Canada. I had blown a transaction. Lemme see it... OK, sorry, that is what the spec says. No, these have to be handled this way, the debit goes over here and credit goes over there when X is Y.

Now as usually happens in these deals I still did not know anything about accounting but some of the words had taken on glimmers of meaning and I asked, But isn't this the same as whatever? And then I threw around some terms and said that from my poor understanding what they were suggesting seemed wrong. Nope, that is how we do it. OK. Bye. Bye.

And now comes the good part. I invoked Tilton's Lemma -- not because Kenny could not see the legitimacy of their accounting, though that encouraged me, but because of something else I found to be strangely compelling: my code would not do what they wanted.

I tried to see how I could patch in the exception and there was no way. I would have needed action at a distance and Einstein had already spoken on that. What was needed was for the microcode to behave differently based on information (X and Y) available only at the top level in which the higher level transactions were being identified and their handlers invoked. The only way to give Canada what they wanted was either to add a parameter to everything in the call chain (which I would have called woof-woof, I think) or even more elegantly to set a global variable at the higher level that this one microcode handler could watch out for. Ewwww.

Now please do not think I am posing as a person of integrity or principle or anything because I am the most miserably pragmatic soul you can hope to meet, but I got back on the phone to Canada.

"Marcia, Kenny. My code does not want to do what you want, eh? Your spec is pretty clear on all these things and just from the sound of it it seems the way I am handling it now is right anyway. I can kludge things up to get what you want, but I will hate myself in the morning."

Marcia said she would talk to Bernie, her VP, and get back to me. She did not do so for so long that I forgot about it and to her credit she did not even have to make the call but she did.

"Kenny, Marcia. I spoke to Bernie. The way you are handling that transaction will be OK. We can live with the way you are doing it."

Tilton's Lemma lived on. Somehow clean design (starting I freely acknowledge with a great spec) had flushed out a faulty business practice. The users had indeed offered an RFE discontinuous with all their other stated requirements, but it turned out to be such an egregious error that they agreed to fix it. (To their credit again!)

Why does this war story give me goose bumps? Yes, I need to get out more. And OK, maybe it is not such a big deal. The careful architecture forced on us by having to program these godawful machines makes it easy for programmers to see inconsistencies the business experts miss.

I just get a kick out of it -- a sense of God or Plato or the Tao throwing their weight around as long as we take design seriously and hop on the phone over such things.

TLOP: The Worst Thing You Can Say About Software Is That It Works

2008-03-22T12:11:00.006-04:00

I walked out of my cubicle, turned left, went up one maze turn, turned into the next cubicle. This was clearly a while ago, before email and even before I had reached the point where I would phone a guy ten feet away in case they were not there so I could leave voice mail.

I wanted to tell Mike that I had enhanced the julian date function he had shared with me so he could replace his with mine instead of the system growing a multiplicity of julian date functions.

Mike was another contractor on some godforsakenly dreary application being developed at cruising altitude in some megabank, a great guy always joking well and laughing about everything including his obsession with the real estate scheme infomercials he apparently watched non-stop in his free time.

I told Mike I had snazzed his function up nicely, generalizing it or adding some whiz-bang feature or something, and mentioned he could now modify his program to use the new version.

Something I need to get across to My Cherished Reader is the utter and profound honesty of Mike's reaction. Indeed, sufficient googling may well surface Mike's blog entry detailing in turn his astonishment and horror at what I had proposed. For when I finished there was a brief moment of incomprehension and then a dawning and his eyes grew wide and his jaw dropped and then a huge smile of wonder spread across his face, his palms rose in supplication for any possible justification of my madness and at last words came to him.

"But...but...," he stammered for effect. "My code already works!"

It was a grand moment, two solid technologists sincere in their craft standing face to face with not one scintilla of common ground on the simplest of questions.

Mike was reeling at the idea that a working module should be touched. He repeated his encomium ("It works!") and then with no offense intended and with solid good-feeling camaraderie just started laughing at me.

Meanwhile I knew without doubt that swapping in my version would likely take one minute (what's that? about sixty seconds?) and that application systems should be kept as simple as possible (and so not have two julian date functions (can you say Y2K?)).

I shrugged -- it was a small matter -- and retraced my path back to my cubicle before the pheromone trail could grow cold. Well, this is my blog, so Mike was wrong. Tilton's Law:

The worst thing you can say about software is that it works.

I think I can prove that with geometry: if a pile of code does not work it is not software, we'll talk about its merit when it works, OK? Therefore to say software works is to say nothing. Therefore anything substantive one can say about software is better than to say it works. QED. (That last bit is the geometry.)

So what else besides that it works can you say about software, especially about applications written in tall buildings? Well, unless one is banking on the relatively high probability that the project will get canceled before ever it sees light of day, the software will eventually be asked to do new things or to do old things differently.

So the software must not be brittle and must not have any more built-in assumptions than necessary and where one builds in the assumptions it would be nice if they were abstracted out of the core so that new requirements can be handled as gracefully as possible.

Note that new and changed requirements come along not just after a successful deployment but also during development, perhaps even more so because only when users see software do they realize what they wanted. They should do a study: I bet requirements change more just implementing the software than they do ever after. Brittle, inflexible software can be so hard to change during development that projects never make it out the door no matter how well the software "works" on the original requirements.

Reminds me of another war story. Schedule "B".

The law on tax shelters had just been changed eliminating the bogus write-offs those things produced and some financial establishment needed some contractor to write the software to generate statements for clients detailing exactly how badly they would be hosed by the new law. Apparently they had tried it in-house and the guy assigned to it had quit after working on it for a couple of months to no avail. Jobs were everywhere back then in the Eighties, why not?

God forbid I should digress: it was a fun interview, really nice group and nice manager. The best point was when he asked His Kennyness what was His greatest weakness. Neat question!

Lemme see...no...no...no... I was looking at the floor going through coding, debugging, DB design, and it was taking a while which I thought was pretty funny and I am sure they were all enjoying it but I really wanted to come up with something and then it came to me.

"Testing!" I almost yelled. "I am terrible at testing!"

Whew! Reminds me of the time I had a little team (this was before people realized I should just be chained to an out-of-the-way workstation and hosed down and thrown food as needed on big projects) and was assigned a junior nooby who asked me the same question about three times and who hated me because apparently Hell hath no fury like a woman gently suggested she might take notes, which was great (that she hated me) because she was so useless we just had her testing my code and believe me no software has ever been subjected to such merciless, sustained, and deliberate abuse as was mine. Every twenty or thirty minutes we would hear this triumphant rejoicing "Ha!" explode from her cubicle as she found another bug and imagined another dagger plunged between my ribs when in fact I was loving the whole thing because to me testing is just God and Abraham and Isaac all over again. Where was I? Schedule "B".

Anyway, the task went very well. The manager (another Mike) gave me a very clear spec and I did my usual structured architectural blah-blah-blah and the users were very nice and called me every once and a while with RFEs. Bringing me to my point.

Mike was a great guy but also button-downed and into process and told me not to listen to the users, make them ask him and he would tell them, No. So I was always very nice with the users and said I welcomed their thoughtful consideration and input and could barely restrain myself from just dashing it off right now if I could just put them on hold for a second and I will indeed start looking at the matter while it would help very much if they could just run this great idea past Mike, he's a bit of a stickler for protocol, terrible bother I know, hate to ask this of you.

OK, they would say.

Then I would put down the phone and look at the code to see what was needed. Oh. Scratch, scratch, poke. Done.

Then I would go into Mike's office and tell him the users had asked for X and he would make a face and I would say I had already made the change if that would affect his decision.

Mind you this did not happen because I consciously planned for change, I just worried about things besides whether the code worked. Simple architectural tidiness went straight to the business bottom line: the users got exactly what they needed at no extra cost to the enterprise. How does simple clean design make RFEs easy to handle? Tilton's Lemma:

Refinements to requirements cannot vary fundamentally from the original.

It is possible for users to say, Oh my, you have the runners running the bases clockwise? We wanted them going counterclockwise. It is not possible for the users to come back and say, We wanted them going first to second base, then over to first, then to third, and then home. Some fundamental force of nature constrains how we can screw up and keeps it within a metalevel such that refinements to requirements will never break code all that badly.

[Keep your eyes open for the "We Can Live With the Way You Handled That" mind-bending illustration of the power of clean design.]

The flipside war story was in a different tall building working on a system so badly designed that after a certain point it pretty much stopped evolving at all.

I walked up to my manager and said I had an idea on how to improve something I was working on for her so that it could be used for other things.

"No, no, no," she said. "I just need this one thing working before Thursday's meeting so I can score one on the users. They are out to get us."

Ah, systems development as an act of war. The system they had developed was such a horror that the only thing that ever moved it forward was a team of very well-paid outside contractors who managed against all odds to add new functionality, pausing from their toils every several days when one of them would take a design two-by-four across his skull from said system and lean back and poll the others on how badly each felt the system hindered them, an order of magnitude generally being settled on as no exaggeration however often it usually is.

The IT department ended up with a bunker mentality lobbing back hand grenade code releases to suppress fire from users strafing us with gratuitous RFEs as the organization staggered towards collapse and acquisition, and it being a financial institution to no small degree dependent on how well it processed information I cannot help thinking this system was to blame.

But it did work.

Oh, sorry. Lisp? Well, the more powerful the language the more time one has for that much easier a job of designing an application as opposed to banging out whatever source code passes the unit tests. And when it comes to separating specifics out into a configuration area that drives a more universal engine... can you say DSL? Sher ya can.

Tilton's Law: Solve the First Problem

2008-03-10T18:21:00.004-04:00

This was such a weird project. Scheduled for five days altogether. My friend from the clinical drug trial venture was also a tech recruiter who got me about half my tech jobs over the years and this one was a real throwaway.

What we had was a mid-80s start-up in the educational software game producing exactly the kind of mind-numbing drill and practice software that was supposed to revolutionize education because Look, Ma! We used computers!

Now they were stuck on some software problem and needed help fast. Their stack was Tandy, Cobol, and some micro database package. My skills were Apple, Cobol, and ISAM and in those days that was a deerskin glove fit so off I went for a mutual look-see.

I was on the beach, why not?

The next morning I am walking up to an apartment building where this enterprise had wedged itself into what was meant to be doctor's offices. Inside I sit down with the top guy in his office and the entire company joins us.

The staff unleashes a thirty minute nightmare tale of software crashes, dysfunctions, anomalies, and disrepair as each person takes turns reciting some utterly bizarre malfunction of the application, all with the database software as the likely culprit. It was a tag-team misery report, a through the looking glass panoply of software non-determinsim. It was wonderful.

A half dozen times I formulated "Explanatory Guess X" only to hear in the speaker's next sentence that they had thought it might be X and but no luck. I mean it was really wonderful and then finally it ended. My head was spinning.

"Have you worked with the Tandy OS," the manager asked.
"No."
"Cobol?"
"Yes, but it does not sound like Cobol is your problem."
"No. I don't suppose you have worked with this DBMS?"
"No."
A pause.

"Can you help us?" See straw. Clutch,

I have no idea what to tell them.

"Is the DBMS any good?", I recover enough to ask.
"I checked it out pretty well. It got great reviews, it is supposed to be the best."
I look down at my shoes.

The contract was for five days. The longest any single glitch had stopped me was for five days. Do the arithmetic.

"Yes," I say.

It took seven. They paid up front for the first five, never paid for the last two probably because they did not have it or maybe because of the way things went. You'll see. And I am surprised it came to seven days, I only remember one or two. I never ran their software once and I do not remember even touching a computer. Here is what happened.

After signing on I took home the manuals for their DBMS and a listing of their schema definition. It took maybe a day to decide that everything looked right. The next day I ask Tom the programmer how hard it would be to just initialize an empty database and start over entering the data.

"Easy", says Tom.

Welcome to Tilton's Law: Solve the First Problem. They had described to me twenty distinct failures and that was too many for me, I am not smart like you guys, I cannot just figure these things out in the shower.

I wanted to turn the software off and turn it back on with a clean slate and see what went wrong first and stop right there. I just wanted to see what went wrong first and fix that. I suspect that needs no explanation, but what am I doing up on this soapbox if I am not going to explain these things?

Here goes. Once upon a time my sleazebag ward politician buddy and I were cruising the singles bars back when they had such things and he got nicely eviscerated by a woman we were chatting up. My buddy had said something cynical and she had challenged him on it.

"Oh, I have compromised my principles a few times," he conceded with a sly grin.

"You can only compromise your principles once," she replied. "After that you don't have any."

Software is the same. This stuff is hard enough to get right when things are working nominally, but once they go wrong we no longer have a system that even should work.

Back on the project, the next day I get a call.

"Bad news," Tom says. Uh-oh.

"What happened?"
"Same thing. Mary was entering the 118th record and the program crashed."

I pretty much fell out of my chair. Somewhere in the thirty minute firestorm of issues I had heard the number 118.

"118 sounds familiar."
"Yep," Tom moaned inconsolably. "That's what happened before. Sorry, no difference."
I was doing cartwheels.

"Tom, how hard would it be to write a program to just write out a couple hundred records, just put in dummy data, 1-2-3-4-5...?"
"That would be easy."
"Awesome, do that and let's see what happens in batch mode," says me.
"OK."
"And reinitialize the DB first, OK?"
"OK."

The next day I hear from Tom. Sounds like he is calling from the morgue.

"Bad news, Kenny."
Oh, no. It worked.

"What happened?"
"Same thing. The program wrote out 118 records and crashed. Sorry, Kenny."
Oh, yeah, I just hate easily reproducible errors. Not!

"Listen, Tom, let's try making the buffer allocation bigger."
"OK."

The next day, "Bad news. Same thing."
I am icing the champagne; this is one solid, reproducible bug. But what about the others?

"Tom, remember the first time this thing crashed, before I came on board?"
"Yeah."
"Did you start over from a fresh database or just resume working on the one that had been open when the DBMS had crashed?"
"We just continued working with the same DB."
"Oh. OK."

Tilton's Law (Solve the First Problem) had been broken as badly as broken can be. A DBMS had failed while writing data and they had tried to continue using the same physical DB. This transgression is so severe it almost does not count.

Normally Tilton's Law refers to two or three observed issues that do not necessarily seem even to be in the same ballpark. The law says pick out the one that seems most firstish and work on that and only that until it is solved. The other problems might just go away and even if not the last thing we need to do while working on one problem is to be looking over our shoulders at possible collateral damage from some other problem.

Two minutes later I am on the phone to DBMS tech support .

"Hi, we're reliably crashing after adding 118 records in one sitting."
"Yes, that is a known problem."
Oh. My. God.

"Would you like us to send you the patch for that?", she asks.
"That would be lovely."

This being before the advent of the Interweb we confirmed our mailing address and asked for it to be sent out ASAP and overnight delivery. But we are not done yet. Tilton's Law or no, all I have solved is P1, the first problem.

"One more thing," I say.
"Shoot."
"If we continue working with the DB after this crash..."
"Oh, no. Don't do that. It's hopelessly corrupted at that point."

Were some of the other issues unrelated to the first crash? I will let you know as soon as this test I have running to solve the halting problem finishes.

Meanwhile, the conversation had suggested how we might get them up and entering data now. Apparently we were crashing because of a bug that surfaced when more than so many records were being held in the buffer before being written out. We had tried making the buffer bigger, only making things worse.

"Tom, we can wait for the patch, but I have one last idea in mind that might get this thing working for you. Want to try one more thing?"
"Sure."
"Try making the buffer half the size it was when we started."
"OK."
A few minutes later he comes back.

"It works now."
"Yeah, baby!"
"I had it loop to one thousand. No problem."
"Cool. Let's tell the others and go get drunk."

Nope. Something is wrong. Tom is just standing in the doorway all deer and headlights.

"Can I ask you something?", Tom asks quietly.
"Sure."
"I do not understand why making the buffer smaller made the program work."
"Well there was this bug that had to do with being unable to keep more than so many records in memory and with a smaller buffer the software did not try to keep so many in memory."
Long pause.

"OK, but why does it work now?"
Hmm.

"Maybe 118 multiplied by the record size is more than 16,384 and somewhere in the DBMS logic there was an integer overflow so the problem does not come up if the cache is smaller and the software flushes the cache before it gets to 16,384."

"All right," says Tom "But I do not understand why we make the buffer smaller and now the software works."

This was surreal. I try a different tack, a really dumb one, but sometimes when a grizzly bear has your back to the wall all you can do is tap dance.

"Look. There are multiple code paths in an application, right? Every conditional is a fork in the path. A bug exists in some branch or other out of all the code paths, right? By changing a fundamental parameter we send the code down a different code path. Avoiding the bug."
Pause.

"I just don't understand why making the buffer smaller makes the program work."

Then it came to me. I was Dr. Chandra in 2010 trying to get Hal to fire the rockets, and Tom was Hal stuck in a Mobius loop unable to resolve my understanding of the confusion with his confusion of the understanding.

"I don't know, Tom," I say. "I don't know why it works now."
Tom nods.

Suddenly Mike, the project lead, appears.

"Kenny, Tom. In my office. Now."
Whoa.

"OK, this has to stop. Kenny, I am paying you to solve this problem and you have Tom doing all your work. He has his own work to do. From now on you work on this problem and Tom you do what you are supposed to be doing. Have I made myself clear?"

Remember in Annie Hall when Woody Allen turns to the camera and asks, Why can't real life be like this?

"Actually...I think I'm done."

Leaving Mike and his facial expression frozen in spacetime, I turn to Tom with raised eyebrows for his assent and Tom nods. I turn back to Mike, who no longer knows where he is.

"It turns out this is a known bug. You'll have a patch tomorrow or the next day. In the meantime we found a workaround and you are up and running. Mary can start entering your data, um, now."

Mike recovers.

"So basically I am sitting here making a complete ass out of myself?"

Good for him. We all had a good laugh, shook hands and I was on my way and Tilton's Law of Programming was reaffirmed: Always solve the first problem. The corollary: there only ever is the first problem.

Kenny and The Firing Squad, Episode II: COBOL Has Macros?!!

2008-03-10T04:40:00.006-04:00

[[If it is good enough for George Lucas (random release of episodes) it is good enough for me.]

Roll the trailer: Kenny hanging with one hand onto the rear axle of a runaway Cobol stagecoach headed straight for a cliff off which a team of otherwise sensible horses soon will be jumping reaching with his other hand to his ankle sheath and pulling out... COPY...REPLACING!]

Well, someone on comp.lang.lisp made a joke about Cobol and Lisp so I thought I would tell this twenty years later still astonishing cobol-and-lisp-separated-at-birth? war story within a war story about how COPY...REPLACING -- no, that is real Cobol , not a Cobol joke -- ended up taking away the award for Best Supporting Language Feature.

A war story within a war story, with the same theme because I guess they shared a central figure, a young project lead we will call (...thinking of a name...) Reggie. That is good. I got it from Regular Guy and I want to make clear that Reggie was a good guy with a good sense of humor, someone I would party with any time, a young guy his manager (another good guy... look, all these people working in tall buildings are good guys tolerating loose cannon per diems like me to get their code written but not inviting us to the company Christmas party (in the cafeteria -- no loss -- I would have showered had I known) actually causing a boycott by one great guy who could not believe I was left sitting alone at my workstation) ... a young guy his manager trusted implicitly I am sure because of Reggie's technical triumphs past.

The outer story is that I was not even supposed to be working on the task from which I was about to be fired for incompetence. I had not even interviewed with Reggie. (Was that the problem? Hmm.) Reggie had gone to an RDB training class and done his first relational schema and an in-house consulting team had deemed it faulty. Reggie's manager backed him so I was brought in as an out-house... hang on... external consultant to provide a second opinion.

I agreed with the in-house team but Reggie would not budge and the manager still backed him (and I need to admit no one was able to make Reggie see his confusion which must be a story in itself because he was listening to us, he really was a good guy). The schema stayed, and a stage was now a phase:

Reggie's mistake was that he saw that two drug trial business objects (phase and stage) both had a one-to-many relationship with the sample biz object, so he treated stage as a phase. But a stage is not a phase. In Phase IV of testing the samples from Phase III were divided into different stages.) But Reg was fixated on that one abstract commonality and could not get past it.

So? Going away lunch for Kenny? Maybe just handshakes all around since I had only been there a couple of weeks? Nope. It turns out I was there for another reason besides my brilliant mastery of relational DB design: they needed someone to grind out nineteen nearly identical query screens which can best be described as exactly what 4GL packages provide at the push of a button.

Feel free to groan.

Yep. Nineteen screens, each aimed at some node in the DB hierarchy. A few fields at the top constituting key fields and selection criteria, while below was a scrolling list of matching items, each column showing some attribute. Select a row, hit the zoom key and.... sorry, could those of you still awake please nudge the ones who are snoring?

The stack was VAX/VMS, COBOL, TDMS for forms management, and I forget what RDB. Based on the nineteen identical screens take one down pass it around eighteen screens to code on the wall functional spec a child of three could see some template approach was in order, if only through the miracle of cut and paste. But I had been around long enough to know how fast the Red Sea can collapse when after cutting and pasting nineteen screens the user comes along and says could these screens work just a wee bit differently?

Now it happens I had been on a project run by a team of English yobbos who had nothing but contempt for us the few American cowboys (they called us) hired to fill some chairs but they were great fun and pretty good and we had together discovered that Cobol not only had copy-replacing but it could even replace part of a name. I forget what it took, but we found just one letter in the ASCII character set (pretend it was the octothorpe) that lets us code:

perform edit-#fieldname.

...in a template and then (guessing at much of this syntax):

copy "edit-template"

Well, one was all we needed, so away we went and life was good and now here I am staring at nineteen screens and I forgot to mention Cobol's MOVE-CORRESPONDING which could populate TDMS data records from DB data records if I was bright enough to name things the same and stand back Argentina! Here come the screens!

I told Reggie what I was going to do. He said, Fine. I said it would help knowing which of the nineteen screens was the nastiest (OK, they were not all that identical). He got it. Fourteen, is a beast. It pulls in data from two yadda yadda. Off I went.

You all know how it goes. I spend a week on screen fourteen, not because it takes a week but because I am building a framework, with one eye on the other eighteen screens sitting in the corner. Hell, I even built an IDE.

The command language for VMS was DCL, a gorgeous little thing. This was all before the multi-window days of VMS. Things got a little tedious as I bounced from editing the TDMS form to the Cobol template to the screen-specific Cobol copy-replacing the template to testing the thing, so I wrote a DCL script that just needed to know on which screen I was working and what I wanted to do and then did The Right Thing. When I exited I fell back into the script, which asked what now? Way cool and productive.

Where was I? Oh. You know how it goes. With screen fourteen working I launch into screen one, hoping it Just Works because Reggie and I had picked out A Beast. Ah, not bad. Screen two. Check. Screen three. Here's a twist. Scribble scribble... good.

Screen four. Excuse me? Two lines of info in order to display one detail? Ouch, did not see that one coming. Bang bang hammer thwack... OK. And on we go. Sure, I still had to build a TDMS form for each screen, but I will live. Or so I thought.

It is Monday of week three and Reggie is standing at the door to my windowless little room.

"Morning."
"Morning."
"How's it going?"
"Great."
"I ask because I haven't seen anything yet."
"Sorry?" Hairs on end.

In five years of programming including three as a pricey independent consultant I had yet to hear one discouraging word from a manager unless you count the time the guy (a very big guy) stood over me beet red literally spitting "Sh*t! Sh*t! You didn't test it, did you?!" as my program gleefully copied only every third record in its first production run because I had left in the bit that did that to make tests run faster. Another day.

"Well it has been two weeks and I haven't seen one screen yet."
"I told you. I am doing all nineteen, not one."
"That's all well and good, but I will be at a class all week, won't be back till Friday, and I need to see something by then."
"Fine." Not sure if you can see the smoke rising from my collar.

It is now Friday noon. Kenny has logged no overtime, but Kenny has had his game face on for four solid days. Co-workers discovered they could rewarm their coffee faster than the microwave if they just put it in the same room with me.

Kenny has little hope of a Hollywood ending, but suddenly Reggie is standing in the doorway, right where he was when he hung the sword over my neck.

"How's it going?
"Good. How was the class?"
"Good."
Silence.
"So where do we stand?"
"With the screens?"
"Yeah."
"They're done."
"They're done? All of them?"
"Yep."
Silence.
"Well, if that's true, that's great."
Withering look. And the fun has only begun.

Late Monday Reggie swings by with a list of trivial bugs and issues.

"Thanks."
"I really like the way it works."
"It's your design."
"Right," and reaches over his shoulder to pat himself on the back.
Did I mention he was a good guy?

"Could you print out the source code for me? I'll read it on the train."
"Will do," looking forward to the praise and astonishment soon to be heaped on me.

Tuesday. My turn to stand in the door of his office.

"Kenny, I was trying to read your code on the train home."
Uh-oh.

"First I'm up here, it says perform this, I have to flip down to find it, then there it says perform that, I'm flipping down again, then I flip back up and it says perform something else and I am flipping down again. Then up then down then up again. What's up with that?"

"It's called structured programming. It's in all the books."

I forgot to mention that by this point I had already given notice. I explained that I thought I was signing on for a short DB design review, not a job in a body shop. They reacted with all the chagrin of the monks at learning Ace Ventura was leaving the temple.

"Yeah, well that's all well and good, but if I cannot understand your code we won't be able to use it."

Well, I was right about one thing, only it turned out to be the garbage heap. By now I think I have made clear to my Honored Reader that I do not consider myself an exemplar of human social interaction including especially the delicacy required by the workplace, but what I did next disappointed even myself but felt great because it was so perfectly honest: I slowly closed my eyes, leaned my head against the door jam, and just stayed that way.

Reggie could not help laughing. Did I mention... yeah, I did.

"All, right. Look. We'll do a code walk-through with Roger [his co-lead] this afternoon, see if we can figure this out."

"Fine."

This afternoon. Twenty-thirty minutes. Copy-replacing. Here is the template, here is a usage. The usage has all and only the stuff specific to that screen. The template. blah blah blah. This is here because of screen eight, two lines to show one detail. If you get other screens like that use eight as your starting point. Oh, this is because of fourteen, that special XYZ requirement. Yadda yadda yadda but there was one strange thing and I was deeply worried.

Neither of them asked a question. Ever. Not one. Have I conveyed the smallness of the questioning? This cannot be good, but I had no more to say.

"Any questions? Reg?"
"No."
"Rog?"
"No." Pause. "Pretty obvious, actually."

And then silence.

Tilton's Law of Programming: Fear No Evil

2008-03-09T05:42:00.005-04:00

We just had this exchange on comp.lang.lisp:

srdjan.m wrote:

On Mar 9, 12:01 am, Ken Tilton wrote:

danb wrote:

On Mar 8, 10:59 am, "srdjan.m" wrote:

(defun test ()
(let ((x '(1)))
(not-ok x)
x))
CL-USER> (test)
(1 C)
CL-USER> (test)
(1 C C)

I really do not understand why let is not initializing x

quote (the single quotation mark) doesn't create a new list, and of
course nconc alters x in place. So you're repeatedly binding x to the
same list and appending 'c to that list.

Incidentally, this is the universal beginner's question. It's been
asked at least three or four times in the last few months. So you
have plenty of company :)

IIANM somewhere in here is either a list or a pointer to a list of these
common nooby coding gotchas:

http://www.cs.cmu.edu/Groups/AI/html/faqs/lang/lisp/top.html

kenny

Indeed there it is http://www.cs.cmu.edu/Groups/AI/html/faqs/lang/lisp/part1/faq-doc-4.html.

Oh my: "Destructive operations, such as NCONC, SORT, DELETE, RPLACA, and RPLACD, should be used carefully and sparingly. In general, trust the garbage collector: allocate new data structures when you need them."

That's nuts! It sows fear and trembling while offering no actual guidance as to when to use destructive operations and when not.

"Use them when you need them." I wonder what the next question will be. (Hint: "When do I need them?".)

As for using them sparingly, what on earth does that mean? What if I "need them" twenty-five times, should I spare a few of them because twenty-five is too many?

Technology is like an ill-tempered dog: never show it fear. If you are not sure how X works do not start inventing weird usage rules unrelated to how the technology works hoping they will somehow keep you safe. You will end up worrying a lot, not get the most out of your tool, and still get bit in the ass.

Instead, take a minute or hour to find out how X works. In this case, there is nothing "in general" or "sparingly" or "as a rule" about it: if you own all the list structure on which you are about to operate, always use the destructive variant. If not, never use the destructive variant.

The non-excepting exception is when the function you are writing is meant to be destructive, in which case the caller is responsible for using it as they would any other destructive function.

So when do I own structure? When I have created it, or called functions that have created it for me. When do I not own structure? When it has been passed to me.

Suppose we have a silly function like this:

(defun do-filter-and-flatten (lists do test)
(apply 'append
 (delete-if-not test
   (mapcar do lists))

mapcar generates fresh structure, so I can use delete-if-not instead of remove-if-not on the list produced by mapcar. But the lists themselves -- the members of the input parameter lists -- came from somewhere else and might be in use, so they cannot be touched. Note that the lists structure itself came from someplace else but does not come into play in this example because we begin by effectively copying that structure while mapcar-ing the do function.

We can turn that around to drive home the point, by applying the filter first (and getting a much different function):

(defun filter-do-and-flatten (lists do test)
(apply #'append
 (let ((tmp (loop for list in lists
                  when (funcall test list)
                  collect list)))
   (map-into tmp do tmp))))

We cannot touch lists (via delete-if-not) because it has been passed to us, so we test and collect. But now we own the returned cons cells bound to tmp and are free to whack their CARs with the mutating map-into -- which Stanislaw Halik just pointed out to me and which I do not think I had even heard of in thirteen years of Lisping!

But if I may digress, Kenny don't play let, certainly not in the middle of a Grahamanian cascade:

(defun filter-do-and-flatten (lists do test)
(loop for list in lists
      when (funcall test list)
      nconc (funcall do list)))

A final note. What if in this last version we knew that any "do" function would massage every element returning a new list along the way. Could we make nconc the last step? I would not. Sometimes functions like these decide they have no work to do and then return the input list structure. We use destructive functions only when we know we own the structure we will be altering, otherwise not.

Then and only then shall we fear no evil.

[All code above Just Typed In, corrections are welcome (and thanks to Anonymous for further reminding me that the non-destructive remove-if-not cannot be assumed to copy -- it might return the input list untouched).]

My Biggest Lisp Project

2008-03-08T05:36:00.006-05:00

Someone asked how much Lisp I have really done. I am building a resume these days getting ready to look for some Lisp work, so I thought I would kill two birds with one stone and write up my experience as the architect and lead developer (out of two, for the most part) of a clinical drug trial management system.

Over a couple of years we built a system consisting of eighty thousand lines of Lisp, having probably thrown away another fifty thousand along the way. We were in the classic situation described by Paul Graham in On Lisp: not knowing exactly what program we were writing when we set out on our mission. We also used C libraries for: writing and reading 2D barcodes; forms scanning; character recognition; generating TIFFs, and probably a couple I am forgetting.

The application was a nasty one:

-- capture clinical drug trial patient data as it was generated on paper at the participating physician's site, using scanner and handwriting recognition tools;

-- validate as much as possible with arbitrary edits; such as complex cross-edits against other data;

-- allow a distributed workgroup to monitor sites and correct high-order mistakes;

-- track all changes and corrections; and

-- do everything with an impeccable audit trail, because the FDA has very strict requirements along these lines.

Making things worse, doctors are often as sloppy about trial details as the FDA is strict about having the rules followed. But drug companies cannot run trials themselves because the FDA demands that investigators be independent to avoid conflict of interest. Getting compliance is tough, and that is the opportunity we were targeting -- more better compliance through automation at a granular, near real-time level sufficient to give drug companies effective oversight over investigator performance.

The stakes are tremendous. Blockbuster drugs can earn millions of dollars a day but only while under patent protection. Unfortunately, patents must be acquired at the start of the trial process, which can run for many years. A third or more of the revenue-rich patent life is spent just getting to market. Big snafus in trials can force months of delay with an opportunity cost of millions a day.

When I got the call from my good friend who was the angel and visionary on this project, he was two years in with not a lot to show for it and his last top developer had just given notice. I went in to hear what they were up to and do an exit interview with the dearly departing.

The business plan was to score big by handling hundreds of trials a year. This would be especially tough because every trial was different. Each involved a custom set of forms to be collected over a series of patient exams. These forms varied from exam to exam. Business logic dictated validation of the forms and how the trial was to run and varied from trial to trial as dictated by what is known as a trial protocol.

When I heard all this I knew we would have to find a solution that did not involve custom programming for each trial. The application would have to be configurable without programming, by a power user trained in the software. If Lisp is the programmable programming language, we needed a programmable application.

Later I learned that competitors in our space had half our functionality and could not handle more than fifteen trials a year and were not profitable. They attempted what they called a "technology transfer" to the drug companies, translating as "we cannot scale this approach but maybe you can". Hmm.

The departing guru showed me what they had so far, which was a system built in Visual Basic with an SQL database. First came the demo of the interactive module, a pure mockup with no substance behind it. Then he showed me the scanning and forms recognition tools in action. He printed out a form built using Word or Visio, scanned it back in, then opened the JPEG file in a manual training tool that came with the recognition software. Field by field he painfully showed the software where each field was, what its name should be, and whether it was numeric, or alpha, yadda yadda.

Ouch. The process was slow and created a brittle bridge from form to application. Worse, these forms might be redesigned at any time leading up to trial commencement in response to concerns from external trial review panels and they can change during a trial in response to field experience. At any given time multiple versions of the same form could be in existence (trials at different sites do not start and stop together), so not only would developers be forever retraining the recognition and then modifying the software to know about new or changed fields, but they also would have to keep alive all the multiple versions of form definitions and match them to specific forms as they got scanned back in or opened for editing.

I was already thinking about automating everything and this process was one that had to be automated. I asked the departing guru if the forms recognition software could be trained via an API instead of via the utility program he was using. He looked at me a moment and said, Yes, realizing I think that is what they should have done. I realized he was leaving in part because he was just a systems guy at heart and this was one deadly application problem to undertake.

To my friend's relief I agreed to give his vision a try. Now it was my turn, but I did not consider for a moment switching to Lisp. This was a serious business application and I knew my friend would never go for it. Nowadays with more experience that is exactly where I would have begun, back then I did not even consider fighting that fight.

No problem. I had pulled off table-driven designs in the past using COBOL and the table-driven design was going to be the key to our success, not the language. Making the table thing work also meant we would want a custom language so we could express the table data easily.

Back home, I looked up at my bookshelf for the manuals to various Lex/Yacc tools I had bought over the years. But I did not look for very long. I knew I could at least use Lisp to quickly prototype the language using a few clever macros, while Lex/Yacc were known to be bears to use --and I had only bought the tools, never really played with either in anger. So Lisp it would be to prototype the trial specification language.

Then I noticed something, my latest hobby creation using Lisp and my cherished Cells hack: an interactive version of the New York Times Double-Acrostic puzzle. It had individual boxes for each letter like any good crossword and that is what we needed for the patient data! So I was halfway home, and sure enough after a few hours I was looking at a perfect mockup of the first page of the sample clinical trial we were using to build the software. And my mouth dropped open at what I saw.

A blinking cursor. In the first character box on the page. I saw that and knew we would be using Lisp for the project. Hunh?

Well, we had to print these forms out, train the forms recognition software, scan the forms back in, feed the scan image to the trained software, and then -- wait for it -- let the users make corrections to the data. And there it was, the blinking cursor waiting for my input. I had been so concerned first with getting the form laid out and ready for printing that I had forgotten that I had cannibalized the code from the crossword puzzle software.

We're done! How famous are those last words? Anyway, my next step was to make sure this wonderful stuff running on my Macintosh under Digitool MCL would run on Windows NT, the planned deployment platform. I figured Lisp I could sell, but the Mac? In 1998? Not.

So on my own dime I acquired an entry-level version of AllegroCL and spent a few weeks porting everything to Windows and it was more of an OS port because Common Lisp is an ANSI standard with which vendors comply pretty well. When I saw everything working on Windows I called my friend, took a deep breath, and said I wanted to use Lisp.

"OK," he said.
"And an OODB," I said.
"OK."

Some fight. The OODB was AllegroStore, then the vendor's solution for persistent CLOS. (Now they push AllegroCache, a native solution where AStore was built atop the C++ OODB ObjectStore.)

Why the OODB? I had noticed something while playing with the database design for the system: I missed objects. I had done some intense relational design back in the day and enjoyed it but by the time in question I had also done a ton of OO and I missed it in my schema design. Not only would AStore give me OO in my DB, it would do so in the form of persistent CLOS, meaning my persistent data would be very much like all my other application data. The whole problem of mapping from runtime data structures to the database -- something programmers have long taken for granted -- would go away.

About a year later we had the whole thing working and I was about to take a working vacation traveling first to San Francisco for LUGM '99, the annual Lisp meeting (here is the best talk) and then on to Taiwan for two weeks of work and socializing with friends. I decided to use this time to tackle a performance problem that had reached "The Point of Being Dealt With". The solution I had in mind seemed like it would be fun (more famous last words) and nicely self-contained and not too onerous for a working holiday.

Basically my approach to performance is not to worry about it until I have introduced enough nonsense that the application is running slow enough to hurt my productivity. Then it is time for what I call Speed Week, a true break from new development dedicated purely to making the system hum again. And we had reached that point.

Otherwise the application was wonderful, just as planned. A trial form was specified using defform, defpage, defpanel, deffield, etc etc. From that single specification came beautifully typeset paper forms, the corresponding DB schema, the unrestricted business logic, the automatic training, scanning, and recognition, and screen forms for on-line editing of the data once scanned in.

Life was good, but run-time instantiation of a form was a pig. It occurred to me that each time a page was instantiated the exact same layout rules were being invoked to calculate where all the text and input fields should fall. The forms author merely specified the content and simple layout such as rows and columns and Cell rules used fontmetrics to decide how big things were and then how to arrange them neatly. The author simply eyeballed the result and decided how many panels (semantically related groups of fields) would fit on a page. So coming straight from the source there was a load of working being done coming up with exactly the same values each time. Some timing runs showed this was where I had my performance problem.

What to do? Sure, we could (and did!) memoize the results and then reuse the values when the same page was loaded a second time, or we could... and then my mouth dropped open again. The alternative (described soon) would let me handle the six hundred pound gorilla I have not mentioned.

First I have to tell you about an overarching problem. One prime directive was that trial sites be fully functional even if they were off the network. Thin-client solutions need not apply. So we had to get client sites set up with information specific to all and only those trials they would be doing. And keep that information current as specifications changed during a trial. My six hundred pound gorilla was version control of the software and forms.

Now here is the alternative. What if we instantiate a form in memory, let the cells compute the layout, and then traverse the form writing out a persistent mirror image of what we find, including computed layout coordinates? Business logic can be written out symbolically and read back in because thanks to Dr. McCarthy code is data. We avoid the redundant computations, but more importantly we now had a changed form specification as a second set of data instead of as a software release. Work on the original performance problem had serendipitously dispatched Kong, because now the replication scheme we would be doing anyway would be moving not just trial data in from the clinical sites but also the configuration data from the drug companies out to the sites. We're done!

If it works. The test of the concept was simple. I designed one of the forms using the trial specification DSL (the macros), compiled, loaded, instantiated, displayed, printed it, did some interactive data entry. Cool, it all works.

Then I "compiled" the form as described above, writing everything out to the OODB. Then I ended my Lisp session to erase any knowledge of the original form source specification.

Now I start a new Lisp session and do not load the source code specifying the form. Using a second utility to read the form information back in from the database, I instantiate the form in memory. And it works the same as I had with the one instantiated from the original specifications. Life was good and about to get better.

Round about now we had the whole thing working, by which I mean all the hard or interesting pieces had been solved and seen to run from form design to printing to capture and validation. Work remained, such as the partial replication hack, but the DB design had taken this requirement into account and was set up nicely to support such a beast. That in turn would let us toss off the workgroup requirement by which remote trial monitors hired by drug companies could keep an eye on the doctors. But then came a four-day Fourth of July and I decided to treat myself to some fun.

For eighteen months we had been working with exactly one specific form from the sample trial. It occurred to me to see how powerful was my DSL, the mechanism by which eventually power users would be describing an arbitrary clinical drug trial to the generic trial manager application, and actually build out the remaining forms of the eighteen visit trial. The experiment would be compromised two ways.

First, I would be the power user. Talk about cheating. But we understood the friendliness of the specification language would have to be developed over time as folks other than me came to grips with it, and also that those power users would have push-button access to software experts when they got stuck -- we were developing an in-house application, not a presentation authoring tool for the general public.

The second compromise went the other way, making things harder. As I proceeded through the new forms I would be encountering specific layout requirements for the first time, and writing new implementation code as much as I would be simply designing the forms in power user mode. Regarding this, I imposed a constraint on myself: I would design the forms to match exactly the forms as they had been designed in Word, even though in reality one normally takes shortcuts and tells the user "you know, if you laid it out this way (which actually looks better) we would not have to change the layout code". But I wanted to put the principle to test that we had developed a general-purpose forms design language.

What happened? It was a four day weekend and I worked hard all four days but the weather was great and I a good dose of Central Park each day skating and fruitlessly pursuing romance. (I liked to concentrate on new skaters because they cannot skate well enough to get away from me.) On the development side as predicted I spent at least half my time extending the framework to handle new layout requirements presented by different forms. But by the end of the weekend I was done with the entire trial. Yippee.

I spent a day writing out the pseudo-code for the partial replication scheme and the other guy started on that while I started thinking about the workgroup aspect. It occurred to me that the mechanism for storing forms describing patient visits could be used for any coherent set of trial information, such as a monitor's so-called "data query" in which they did a sanity check on something that had passed validation but still looked wrong. The only difference with these forms compared to the trial data forms (already working) would be that they would not be printed and scanned, so... we're done!

Talk about code re-use. Somewhere along the way I had accidentally created a 4GL in which one simply designed a screen form and went to work, the database work all done for you.

Now I started working on the interface that would knit all this together. GUIs are insanely easy with my Cells hack. Downright fun, in fact. So much fun, so easy to do, and then so foolproof that I missed them almost immediately. The persistent CLOS database lacked Cells technology. Of course. It was just persistent data as stored by the AllegroStore ODB.

But Cells at the time was implemented as a CLOS metaclass, and so was AllegroStore... no, you are kidding... multiple-inheritance at the metaclass level?! That and a little glue and... we're done! The GUI code now simply read the database and showed what it found to the user. When anything changed in the database, the display was updated automatically. (I always enjoy so much seeking out the Windows "Refresh view" menu item. Not!)

For example, the status of form Visit #1 for Patient XYZ might be "Printed". Or -- since one business rule said forms should be printed and scanned in short order so the sytem could tell when forms had been lost -- it might say "Overdue for Scanning". So now the user looks around, finds the form, puts it in the sheet feeder of the scanner and hits the scan button. In a moment the user sees the status change to "Ready for Review".

The beauty of having wired up the database with Cells is that the scanning/recognizing logic does not need to know that the user is staring at an "Overdue" indicator on the screen that no longer applies. Tech support will not be fielding calls from confused users saying "I scanned it six times and it still says overdue!" The scanning logic simply does its job and writes to the database. The Cells wiring notices that certain GUI elements had been looking for this specific information and automatically triggers a refresh. And as always with Cells, the GUI programmer did not write any special code to make this happen, they simply read that specific bit of the database. Cells machinery transparently recorded the dependency just as it automatically propagates any change.

In the end we had taught Cells and the ODB a few kinds of tricks. It was possible for a dynamic slot of a (dynamic) GUI instance to depend on the persistent slot of a DB instance, or on the population of a peristent class ("oh, look, a new data query just came in"). Persistent instances can have dynamic slots in AllegroStore and these could depend on persistent slots, and perhaps the scariest bit: persistent slots could depend on other persistent data. The database was alive!

One change could ripple out to cause other change in the database. For example, "Overdue Form" was a persistent attribute calculated from the fact that a form had been printed but that it had not been scanned. If that status held for a day the database automatically grew a new persistent instance, an "Alert" instance visible to trial monitors who could intervene to see why the clinical site was not taking care of business. When the form got scanned, Cells logic caused the status to move up to "Scanned/ready for review" and the "Alert" instance got deleted. All in classic Cells declarative fashion: "an Alert exists if a document is overdue for one day" takes care of both creating and destroying the instance.

I'll never forget a religious moment I had scanning forms so I could work on the interface's mechanism for correcting scanner or recognition errors. When I called up page two of some form all the fields were blank. I had never seen the scanner/recognition software completely miss a page, but a peek at the log file showed the page had indeed gone unrecognized. (We used 2-D barcodes to identify pages.) Stunned at this first ever failure (out of hundreds of trials) I just grabbed the second page, put it back in the sheet feeder and hit the scan button.

The nice thing about the barcodes is that I could just do that, I did not have to tell the software "OK, now I am scanning page two of form 1 for patient XYZ." The barcode data was in fact nothing but the GUID we assigned to the page when creating it in the ODB, so the printed paper had object identity. :) The other neat thing here is what when instantiating a page we linked it to the version of the form template from which it was derived. This took care of the version control problem created by changing forms -- a scanned page was able to look in the database to find the template from which it was created and scan and open itself. Users never had to say what they were dropping into the sheet feeder or worry about the order. Back to my missing page...

The scanner started whirring, the page went through, and then log diagnostics from the recognition process started zooming by (OK, this time we recognized the page, the last time was truly a fluke) and then even before it happened I realized (Omigod!) what was about to happen.

I turned my eyes to the blank page still up on my screen and waited...Boom! The data appeared. Having wired the database to the screen, what happened was this: the recognition logic simply read the forms and wrote out the results as usual, updating each persistent form field with a value. The screen field widget had gotten its display value by reading that DB field. Cells told the screen field widget to "calculate" again its display value. It was different, so the Cell observer for the screen field generated an update event for the field. The application redrew the field in response to the update event.

That was so cool to see happen.

We're done! Literally. :( Eventually our tiny little operation was never able to persuade big pharma we could handle the grave responsibility of not screwing up trials, even though IBM itself loved our work and worked with us to pitch it to pharma. But that is another story.

OpenLaszlo-Cells Smackdown 2008

2008-03-08T00:19:00.004-05:00

Yes, I pointed this out before on c.l.lisp but there has to be some upside to senility and freely repeating myself is it: an OpenLaszlo introductory video.

Pretty far into the video, after some horrifyingly dirty language like "JDOM" which was all the more horrifying for the speaker clearly believing he was talking plainly (driving home my dinosaurosity and at the same time making me feel a whole lot better about it) we learn that OpenLaszlo includes as one bell amidst all the whistles...wait for it... dataflow! ta-dum!!! Pretty damn transparently, to boot.

Meanwhile over in the comp.lang.lisp archives you'll find the author of the latest Common Lisp tome saying he cannot see the point of my equivalent Cells library and news of my invitation to ECLM 2008 where I have been invited to unravel this profound mystery live and in color as if that will make any difference.

Meanwhile back in the above video we discover the speaker presuming the advantages are obvious to his audience and offering no further explanation beyond pointing it out.

Tilton's Law: Solve the Right Problem. Maybe I do not need to explain dataflow better, maybe I need a new audience.

...sigh...

ps. In the yobbos's defense, I just checked the Laszlo white paper on their product and they do not even mention dataflow explicitly. Just a couple of mentions of layout being "declarative". :)

My Nastiest Macro Ever?!

2008-03-05T19:49:00.005-05:00

[WARNING: Do not try at home. The following code relies on extensions not included here. Plz visit the DSB thread for something you can play with]

It all started because a Lisp/Arc noob "got it" when they saw how I used Lisp destructuring to make some code a little more readable. That in turn started with a challenge to improve some code to write an SQL query or something.

I suggested rearranging the Lisp query spec to be friendlier to the consumer, then used Arc destructuring as the first improvement (and then greatly missed CL format):

(let (tbl key . fields) *sql-template*
   (prn (string "insert into " tbl
           " ( " (let c nil
                   (each (f . rest) fields ;; more destructuring
                     (when c (= c (string c ", ")))
                     (= c (string c f)))
                   c)
          ", " (car key)
          " ) values ( "
          (let c nil
            (each (f . rest) fields
              (when c (= c (string c ", ")))
              (= c (string c ":" f)))
            c)
          ", " (last key) ".nextVal"
          " ) returning " (car key) " into " (string ":" (car key)))))

When the "got it" came in, I decided to dust off my lite version of CL's destructuring-bind (actually done to implement a CL-style defun for Arc since its DEF has only optional args) and really give the Arcers an eyeful.

The functionality we are after, by example:

(let data (list 1 2 7 nil 'p 5)
  (dsb (x y &o (a (+ y 1)) (z 4) &k (p 98) (q (+ a 1))) data
    ;; we want to see identical pairs, cuz next I
    ;; print first a variables runtime binding
    ;; and then its expected value
    (prs "args" x 1 y a 7 z nil p 5 q 8)
    (prn)))

The above should produce "args 1 1 2 2 7 7 nil nil 5 5 8 8". Yes, I apologized to the Arc forum for my lame test "utility". Anyway, here is what I came up with, and even though I have written well over five hundred macros this one just might be the most mind-bending I have done.

First, the final form (probably still buggy!):

(mac dsb (params data . body)
  (w/uniq (tree kvs)
    `(withs (,tree ,data
              ,@(with (reqs nil key? nil opt? nil keys nil opts nil)
                  (each p params
                    (if
                        (is p '&o) (do (assert (no opt?) "Duplicate &o:" ',params)
                                       (assert (no key?) "&k cannot precede &o:" ',params)
                                     (= opt? t))
                      (is p '&k) (do (assert (no key?) "Duplicate &k:" ',params)
                                     (= key? t))
                   key? (push-end p keys)
                   opt? (push-end p opts)
                   (do (assert (~acons p) "Reqd parameters need not be defaulted:" p)
                       (push-end p reqs))))
                  (with (n -1)
                    (+ (mappend [list _ `(nth ,(++ n) ,tree)] reqs)
                      (mappend [list (carif _) `(if (< ,(++ n) (len ,tree))
                                                    (nth ,n ,tree)
                                                  ,(cadrif _))] opts)
                      `(,kvs (pair (nthcdr ,(++ n) ,tree)))
                      (mappend [list (carif _)
                                 `(aif (member',(carif _) ,kvs)
                                    (cadr it)
                                    ,(cadrif _))] keys)))))
       ,@body)))

That hurt. But why is it possibly the wildest one I have ever written? It certainly is not the longest. Let me put it this way: macros are always tricky but most of them only go up to ten; this one goes to eleven.

First, let's look at a simpler example:

(dsb (x &k (y (++ x))) (list 1 'y 2)
   (list x y))

...and the desired expansion:

(withs (u1 (list 1 2)
         x (nth 0 u1)
         (u2 (pair (nthcdr 2 u1)))
         y (bif (assoc 'y u2) (cadr it) (++ x)))
  (list x y))

Come to think of it, next time I have to write a macro this tricky I just might write out the expansion I have in mind rather than try to hold it in my imagination as I code. Lesson learned, although macros normally arise after we have indeed written several long versions and then decide to write a macro, so we do have expansions to stare at. Anyway...

Macro writing does present a challenge in all but the simplest token-replacing cases: while writing the macro we are working in two different times at once, macro-expansion time and run-time. I am writing macro code M (to run at macro-expansion time) whose input is source code S and whose output is code R that will Actually Run at run-time. As the macro author, I start a macro function in mindset M, then use a backquote to start specifying code in mindset R, then use the unquoting comma to jump back to mindset M. It is as if my time machine went haywire and I am experiencing now and the future on alternate blinks of the eye.

When this becomes second nature, btw, you can add Lisp to your resume.

So why is this example worse than usual? How does it go to eleven? Look at the binding of U2 (which for the cognoscenti is my fakery of the unique symbol uniq (CL's gensym) would produce). Omigod! The runtime code R is finishing the job of parsing the list being destructured by taking arguments beyond the last optional position and building them into an a-list to be read by the value forms of each keyword, themselves doing more parsing to decide if a runtime value was supplied or if they should use the form (if any!!!) specified as the default in the parameter list!!!

Aaaaaargghh!!! Maybe this one goes to twelve. I observed on the Arc forum while describing all this that it was a good thing I had written the macro before thinking about it. But there we were, so before closing the thread I decided to mention some simple macro-writing tips:

1. The nice thing about CL-style and now Arc-style macros is that they are just like any other code we write, so we can debug them the same way, e.g, by placing print statements in the M code, if you recall my naming convention. This will be a mind-bender at first, or it was for me anywho.

2. If when you test a macro it dies on the compile of a usage (not the run) your bug is in the M code, not the produced runtime R code. Let's go into some detail:

I can write a macro and have that not compile:

(mac surprise! (f1 f2)
      `(do1 ,f1, f2, ,f3))

... because f3 is undefined. Duh. Easy enough to nail those. Once I fix the macro:

(mac surprise! (f1 f2)
      `(do1 ,f1, (/ ,f2 0))

...I can try it out and come to grief on a reasonable looking usage because the macro generates buggy code:

(surprise! 10 11)

...which should produce a division by zero error by expanding to:

(do1 10 (/ 11 0))

But there is another way to mess up! Isn't this fun! You can have macro code M that compiles OK but has a bug of its own that will be encountered while the M code is running (to produce the R code for the compiler):

(mac surprise! (f1 f2)
      (let x (len f2) ;; golly I hope f2 is a proper list
        `(do1 ,f1, (/ ,f2 0))

That version of the macro compiles, but now:

(def three-two-one ()
         (surprise! 10 11)

...should not compile, Arc should squawk during compilation of three-two-one about 11 not being a proper list.

In summary, when and how a macro usage goes South tells you in which of these many ways possible you have screwed up. Until you develop an instinct for that, this will help:

2. Once your macro itself compiles (not a use thereof), use macex to preview and debug the R code your macro will pass on to the compiler:

(prn (macex '(surprise! 10 11)))

The above may fail if, as we discussed, you have a runtime bug in your macro expansion code M. If you are developing a macro and testing it in your application and get confused as to when the wheels are coming off, take a step back and cut-and-paste the failing usage into the above, not forgetting the quote.

This by the way is a helpful reminder that macros eat symbolic source code, not text strings, and not runtime values, by which latter I mean this next bit should work even though the variables hi and mom do not exist -- at this point again it is all just symbols:

(prn (macex '(surprise! hi mom)))

Use macex also when you simply do not get the behavior you want, when a macro usage compiles and runs but gives the wrong result. It might be that a macro expansion is writing bad code, and it is easier to see in the expanded form of an actual usage than it is staring at the slicing-dicing of a macro function. Come to think of it, we can see that with this example:

(surprise! 1 2)
 -> 1

Puzzled as to why we get 1 instead of 2, we use macex:

(macex '(surprise! 1 2))
->(do1 1 2)

Aha! That is supposed to be do! The do1 is left over from an earlier version where it made sense. Something like that.

3. Divide and Conquer I: first I did DSB just with required params, then with optional params, then with keyword params, then with default values, then with computed default values, yadda yadda yadda, nothing new in this tip but there ya go.

4. Divide and Conquer II: This gets back to tip #1: macros are like any other code, meaning we can break complex chunks out into standalone functions to be tested separately. See that little state machine at the beginning of DSB that parses the parameter list? That is a nice little standalone chunk with a big job to do, let's give it its own home:

(def dsb-params-parse (params)
  (with (reqs nil key? nil opt? nil keys nil opts nil)
    (each p params
      (if
          (is p '&o) (do (assert (no opt?) "Duplicate &o:" ',params)
                         (assert (no key?) "&k cannot precede &o:" ',params)
                       (= opt? t))
        (is p '&k) (do (assert (no key?) "Duplicate &k:" ',params)
                       (= key? t))
        key? (push-end p keys)
        opt? (push-end p opts)
        (do (assert (~acons p) "Reqd parameters need not be defaulted:" p)
            (push-end p reqs))))
    (list reqs opts keys)))

And test:

(prn (dsb-params-parse '(x y &o (a (+ y 1)) z &k (p 98) (q (+ a 1)))))
-> ((x y)((a (+ y 1)) z)((p 98) (q (+ a 1))))

Awesome. Now look at DSB:

(mac dsb (params data . body)
  (w/uniq (tree kvs)
    `(withs (,tree ,data
              ,@(let (reqs opts keys) (dsb-params-parse params)
                  (with (n -1)
                    (+ (mappend [list _ `(nth ,(++ n) ,tree)] reqs)
                      (mappend [list (carif _) `(if (< ,(++ n) (len ,tree))
                                                    (nth ,n ,tree)
                                                  ,(cadrif _))] opts)
                      `(,kvs (pair (nthcdr ,(++ n) ,tree)))
                      (mappend [list (carif _)
                                 `(aif (assoc ',(carif _) ,kvs)
                                    (cadr it)
                                    ,(cadrif _))] keys)))))
       ,@body)))

Sweet! Now we can give Arc a Cl-style defun if like me we are fond of them:

(mac defun (name params . body)
  (w/uniq (args)
    `(def ,name ,args
       (dsb ,params ,args ,@body))))

Applied, we create the CL poster boy for mixed optional and keyword args:

(defun read-from-string (s &o eoferr? eofval &k start end keep-whitey?)
        ....)

Too easy? :)

Somewhere out there some Pythonista believes the sky is falling, that I have forked Arc, and that no one will ever be able to read my code. (a) They are wrong, and (b) this is a wonderfully extreme example in which we really are (yes!) extending the language. If they do not like this, maybe they will like some of the simpler transformations mac allows.

Macro-ro-ro Your Code Gently Down the Screen...

2008-02-28T04:35:00.008-05:00

Sorry about the title but I just feel a little silly having to explain why macros rock so hard when the C preprocessor has been doing them almost as long as Lisp. It just ain't that hard an idea to get. It seems to me anyone writing a sufficient amount of code will notice recurring patterns not susceptible to attack by encapsulation in a function. Consider OpenGL.

With OpenGL, certain things have to happen between a glBegin and a glEnd, and certaing things are not allowed between a glBegin/End pair. One of them being a second glBegin. The poor OpenGL programmer now has one more thing to worry about, and tedious boilerplate to write like a robot (in this case just a one-liner glEnd() but try not to lose the big picture principle, OK Sparky? This is a deliberately simple example). Is there any way to alleviate this drudgery?

Well, quite a few statements will be squeezed of between glBegin and glEnd, cuz OpenGL is no joy to code. We can write a function called do-between-begin-end and pass it a first class function (unless your BDFL does not see the need), but then (in CL) we end up with:

(do-between-begin-end
     (lambda ()
        ...your dozen lines of code here...
        ))

Not the end of the world, but remember, I am just warming up. These macro posts will go on forever. Anyway, with a macro we lose the lambda speed bump:

(with-gl-begun ()
...your dozen lines here...)

What's with the ()? Well, truth is glBegin takes a parameter that says what we are beginning (line strip? triangles? quads?) so I kinda lied, the contrast should be...

(do-between-begin-end gl_quads
     (lambda ()
        ...your dozen lines of code here...))

and:

(with-gl-begun (gl_quads)
     ...your dozen lines here...)

So, yeah, I can write do-between-gl-begin-end and never worry about begin and end again, but I am stuck with a little line noise there.

If we look at a classic example of how Lisp macros hide boilerplate, we see this more egregious after/before pair:

(with-open-file (f "xxx")
   (print "line 1 of xxx:")
   (print (read-line f)))

Versus what I have to write without with-open-file:

(let ((f (open "xxx"))
     (success nil))
    (unwind-protect
      (multiple-value-prog1
        (progn
          (print "line 1 of xxx follows:")
          (print (read-line f))
          (close f))
        (setf success t))
 (unless success
   (close f :abort t))))

The odd thing is that Pythonistas reject macros while fervently embracing tidy code -- and Python code does look very clean -- but all we are doing with macros (so far) is making code neater. Well, no, we are also releaving the programmer of drudgery and making the code more reliable by automating the insertion of necessary boilerplate such as error checking.

Let's turn now to a dead simple hiding of boilerplate, Paul Graham's AIF swiped from the inestimable On Lisp:

(defmacro aif (test-form then-form &optional else-form)
    ‘(let ((it ,test-form))
        (if it ,then-form ,else-form)))

The alternative is:

(let ((temp (big-long-fn a b c)))
    (if temp
       (handle-it temp x y)
      (do-something-else a x))

By the way, if you do not like Graham's anaphoric thing (using "it" as the implicit temporary variable), use my BIF which takes a mnemonic for the intermediate result:

(defmacro bif (bindvar boundform yup &optional nope)
  `(let ((,bindvar ,boundform))
     (if ,bindvar ,yup ,nope)))

And then a more realistic use:

(bif sol (math-problem-solution prb mx)
  (display-solution sol)
 (tell-user "No can do!"))

In this just my first offering on macros I have discussed only the advantages of automatically generating boilerplate. There is much more to come, but I would like to close by reiterating my befuddlement that this even needs explaining, and in the context of the C preprocessor precedent mentioned above:

I just dug into the old C version of my Algebra software for just one out of hundreds of macros, one which iterates over the children of a node (a custom tree struct in my application) in something akin to the C "for" statement:

#define fork(PAR,TMP_KID,NXT_KID)      \
    for ( NXT_KID = (TMP_KID = firstk( PAR))? nexts(TMP_KID):NIL \
     ; TMP_KID  \
     ; NXT_KID = (TMP_KID = NXT_KID)? nexts(TMP_KID):NIL)

The crucial thing to note is that this macro is meant to allow children to be unlinked from the parent during the traversal, so it is necessary to capture the next child after the current child before processing the current child, because parents only see their first and last child as I had things arranged.

So now I could write code like this (instead of something like the macro definition above):

fork( usg, tusg, nusg) {
    kidDetach( tusg); ;; losing the "next child" link
    xInsKid( tpar, pusg, tusg);
    }

Next time we will do something even jazzier.

Ooh! Ooh! My turn! Why Lisp?

2008-02-26T14:10:00.002-05:00

Brace yourself. The first reason you should be using Lisp is a non-reason, an answer to an objection, a negation, a let-down: Lisp is just an ordinary programming language.

I'll wait while you pick yourself off the floor. What a terrible place to start! But it must be said. It is the first hurdle on the road to Lisp. Xenophobia never feels like fear of the strange, it feels like dislike. But one of the best young Lispers to show up at a Lisp-NYC meeting sat down, ordered a beer, and was smart enough to make it his first question: Can you use Lisp for conventional programming? Can it read a database? Can it run a GUI?

Yep.

Lisp at one level of understanding is just a normal high level programming language. Those parentheses look quite different but constitute only a superficial difference from the conventional chicken scratch syntax of semi-colons, braces, full stops, and brackets. When a conventional programmer sits down to program Lisp, they feel quickly at home: Lisp is just another 3GL.

I feel terrible about this. I am supposed to be up on a soapbox preaching eternal salvation and ecstasies glorious and unknown and we will touch on that below but in the end your correspondent is just a simple application programmer and if you check my Road to Lisp you will discover that I (re)discovered Lisp just to find an easier way to create educational math software.

Lispers are accused of religous fanaticism and zealotry -- nope, our enthusiasm is all about getting applications built faster with less pain. Which brings me to the bad news: the next reason I have to offer for using Lisp is almost as boring as the first: Lisp is mature. Turns fifty or so in 2008. In 1994 when it was old enough to be President of the United States it got an ANSI Standard. How stable can you get?

So far we have ordinary, old, and codified and I am losing my audience. Send in the clowns!

Fine, a little pizazz: it ain't just old. The old translates to polish. Lisp did not begin with a standard and it did not begin as a single-implementation BDFL strangulation such as Python or Perl or Ruby. It began as...an experiment?

Two score and ten years ago our foregeeks started from the simple idea that code and data were one and began evolving a huge tool chest of HLL functionality spread across several competing dialects. For some reason they took the engineering so seriously that whenever a nooby picks up one of these tools surprise! surprise! it turns out to work precisely the way they need it to. It seems like magic but it is not. The magic is just a long history of use and careful refinement by very good engineers and was possible precisely because there was no standard and there was no BDFL.

Back to the yawns: most Common Lisp implementations are native compiled. Yeah, yeah, I know, computers are fast and we are all doing Web apps where network latency decides all but I work on a desktop interactive expert system that needs all the horsepower it can get and to tell you the truth when building something like that more horsepower makes more better features possible so isn't it nice to compile down to the metal?

More yawns: ephemeral GC, the best OO model you can imagine, multiple implementations to choose from including free ones.... hey, even I am getting bored. But my Road also reveals that I stumbled onto this staid dependable workhorse only after squandering mucho dollars and hours on half-baked gleams-in-the-eye programming tools that promised a world of productivity and delivered only reboots. At which point boring looks pretty damn exciting.

Left as an exercise is connecting the stable/standard/refined dot with other dots of dynamic interactive languages still emerging from sea and periodically burying their installed bases in volcanic ash. Can you scream Perl 6? Sher ya can. Has Guido finally axed first-class functions yet? It is fun watching languages evolve, but if you will excuse me I really do have an application to get out.

With the dull stuff out of the way we can indulge at long last in some Why Lisp? fireworks.

We'll build slowly, starting with Lisp being interactive, dynamic, reflective, functional, compiled, mature, standardized, has a vast array of library functions and data structures (including the wonderful and ubiquitous list), multiple value return, special variables, CLOS (including multiple inheritance and multi-methods), the MOP, first-class functions and omigod! closures, destructuring-bind, LOOP, lexical scope, untyped variables, typed data, object identity for everything but numbers and characters and now repeat a few of those things a few times to make clear how helpful they are to getting applications built, such as the vast toolchest. Gasp.

I call that starting slowly because everything above can be found in some language. It is wonderful that only Common Lisp has them all, but is there anything that really sets Lisp off? Is there some protein that cannot be mimicked by other invading dynamic language organisms, some shibboleth their tongues cannot form?

More prosaically, is there some quality those languages cannot copy without becoming Lisp?

At ILC 2002 former Lisp giant now Python advocate Peter Norvig was for some reason allowed to give the keynote address like Martin Luther leading Easter Sunday mass at the Vatican and pitching Protestantism because in his talk Peter bravely repeated his claim that Python is a Lisp.

When he finished Peter took questions and to my surprise called first on the rumpled old guy who had wandered in just before the talk began and eased himself into a chair just across the aisle from me and a few rows up.

This guy had wild white hair and a scraggly white beard and looked hopelessly lost as if he had gotten separated from the tour group and wandered in mostly to rest his feet and just a little to see what we were all up to. My first thought was that he would be terribly disappointed by our bizarre topic and my second thought was that he would be about the right age, Stanford is just down the road, I think he is still at Stanford -- could it be?

"Yes, John?" Peter said.

I won't pretend to remember Lisp inventor John McCarthy's exact words which is odd because there were only about ten but he simply asked if Python could gracefully manipulate Python code as data.

"No, John, it can't," said Peter and nothing more, graciously assenting to the professor's critique, and McCarthy said no more though Peter waited a moment to see if he would and in the silence a thousand words were said.

Come to think of it...

Arc!Cells: It's Alive!!!

2008-02-18T20:54:00.004-05:00

In the Cells Manifesto I explained what all the Cells hoopla was about, including the vast array of other people doing the same thing only not as well as His Kennyness.

In the Baby Steps article we saw a primitive exploration of the mere beginnings of a partial implementation.

Well I am through kidding around! Below the fold you will find what should be standalone code for a solid Arc implementation (albeit not as fully splediferous as The Real (Common Lisp) Deal) of vanilla Cells. Also available from CVS.

The furnace example at the end will (a) provoke even more nutty ads for furnace installations (sigh) and provide a rough idea of the magic: make one state change and like so many dominoes other state change follows as effect follows cause, with user-defined observer callbacks invoked so our virtual models can act on the world outside the model, all under a declarative paradigm the academics call reactive programming because a better name like dataflow would not make them sound so... I digress.

Enjoy. Arc code enhancements always welcome, especially where I have reinvented built-in Arc constructs.


;; - - - - - - - - - - - - -  cut here - - - - - - - - - - - - - - 
;;
;; copyright 2008 by Kenny Tilton
;;
;; License: MIT Open Source
;;
;;

;;; --- detritus ------------
;;;

(def prt args
  ; why on earth does prn run the output together?
  (apply prs args)
  (prn))

(def tablemap (table fn)
  ; fns are always huge and then a tiny little table ref just hangs off the end
  (maptable fn table)
  table)

(def cadrif (x) (when (acons x) (cadr x)))

(mac withs* (parms . body)
  ; faux dynamic binding
  (let uparms (map1 [cons (uniq) _] (pair parms))
    `(do ,@(map1 (fn ((save curr val))
                   `(= ,save ,curr ,curr ,val)) uparms)
       (do1
           (do ,@body)
         ,@(map1 (fn ((save curr val))
                   `(= ,curr ,save)) uparms)))))

;;; -------------------- Cells ----------------------
;;;
;;;  A partial implementation of the Cells Manifesto:
;;;      http://smuglispweeny.blogspot.com/2008/02/cells-manifesto.html
;;;   
;;; --- globals --------------------

(= datapulse* 0) ;; "clock" used to ensure synchronization/data integrity
(= caller* nil) ;; cell whose rule is currently running, if any
(= mds* (table)) ;; model dictionary
(= obs* (table)) ;; global "observer" dictionary

;;; --- md -> modelling ----------------------------------------

(mac defmd ((type-name (o includes)
              (o pfx (string type-name "-")))
            . slot-defs)
  `(do
       (deftem (,type-name ,@includes)
           ctype ',type-name
         cells nil
         ,@(mappend (fn (sd) (list (carif sd)(cadrif sd))) slot-defs))
       ; define readers
       ,@(map (fn (sd)
                `(def ,(coerce (+ (string pfx) (string sd)) 'sym) (i)
                   (slot-value i ',sd)))
           (map carif slot-defs))
     ; define writers
     ,@(map (fn (sd)
              `(def ,(coerce (+ "set-" (string pfx) (string sd)) 'sym) (i v)
                 (set-slot-value i ',sd v)))
         (map carif slot-defs))))

;;; --- model initialization

(def to-be (i)
  (do1 i
    (md-finalize i)
    (md-awaken i)))

(def md-finalize (i)
  (do1 i
    (if (acons i)
        (map md-finalize i)
      (do
          ; register instance in a namespace for inter-i dependency
          (= (mds* (md-name i)) i)
          
          ; move cells out of mediated slots into 'cells slot
          (tablemap i
            (fn (k v)
              (when (c-isa v 'cell)
                (= v!model i v!slot k)
                (push (list k v) i!cells)
                (= (i k) 'unbound))))))))

(def md-awaken (i)
  (do1 i
    (if (acons i)
        (map md-awaken i)
      (do ; bring each slot "to life"
          (tablemap i
            (fn (k v)
              (aif (md-slot-cell i k)
                (slot-ensure-current it)
                (slot-value-observe i k v 'unbound))))))))

(def md? (name)
  mds*.name)

;; --- start of cells stuff ------------------

(def cells-reset ()
  (= datapulse* 1) ; not sure why can't start at zero
  (= caller* nil)
  (= mds* (table)))

(def ctype-of (x)
  (when (isa x 'table)
    x!ctype))

(def c-isa (s type)
  (is ctype-of.s type))

(defmd (cell nil c-) ;; the c- gets prefixed to all accessor names
    awake 
  (pulse 0)
  (pulse-last-changed 0)
  (cache 'unbound)
  model
  slot
  rule
  users
  useds
  observers)

(defmd (model nil md-)
    ; any template to be mediated by cells must include model
  name ; used so one instance can find another by name
  cells
  observers)

(def md-slot-cell (i s)
  (alref i!cells s))

;;; --- reading a slot -------------------------

(def slot-value (i s)
  (aif (md-slot-cell i s)
    (do (when caller*
          (pushnew caller* it!users)
          (pushnew it caller*!useds))
        (slot-ensure-current it))
    (i s)))

(def calculate-and-set (c)
  ; clear dependencies so we get a fresh set after each rule run
  (each used c!useds
    (= used!users (rem c used!users)))
  (= c!useds nil)

  ; run the rule
  (let nv (withs* (caller* c)
            (c!rule c!model))
    (unless c!useds
      ; losing rules with no dependencies 
      ; is a big performance win
      (optimize-away c))
    (slot-value-assume c nv)))

(def optimize-away (c)
  (pull (assoc c!slot ((c-model c) 'cells)) ((c-model c) 'cells))
  (each user c!users
    (pull c user!useds)
    (unless user!useds ; rarely happens
      (optimize-away user))))

(def slot-ensure-current (c)
  ; It would be fun to figure out a more readable
  ; version of the next consition. I tried, can't.
  (when (and c!rule
          (or (is 0 c!pulse-last-changed)
            (no (or (is c!pulse datapulse*)
                   (no (any-used-changed c c!useds))))))
    (calculate-and-set c))

  (= c!pulse datapulse*)

  (when (is 0 c!pulse-last-changed) ;; proxy for nascent state
    (= c!pulse-last-changed datapulse*)
    (slot-value-observe c!model c!slot c!cache 'unbound))
  c!cache)

(def any-used-changed (c useds)
  (when useds
    ; So happens that FIFO is better order for this
    (or (any-used-changed c (cdr useds))
      (let used (car useds)
        (slot-ensure-current used)
        (> used!pulse-last-changed c!pulse)))))

;;; --- writing to a slot -----------------------

(def set-slot-value (i s v)
  (aif (md-slot-cell i s)
    (do (++ datapulse*)
        (slot-value-assume it v))
    (prt "you cannot assign to a slot without a cell" i s)))

(def slot-value-assume (c v)
  (= c!pulse datapulse*)
  (with (i c!model ov c!cache)
    (unless (is v ov)
      (= c!cache v)
      (= (i c!slot) v) 
      (= c!pulse-last-changed datapulse*)
      (slot-propagate c ov)))
    v)

;;; --- dataflow --------------------------------
;;; Propagate state change from cell to cell and
;;; as needed from Cell to outside world
;;;
(def slot-propagate (c ov)
  (let caller* nil
    (each user c!users
      (slot-ensure-current user))
    (slot-value-observe c!model c!slot c!cache ov)))

(def slot-value-observe (i s v ov)
  (awhen (md-slot-cell i s)
    (observe it!observers i s v ov))
  (observe (alref i!observers s) i s v ov)
  (observe obs*.s i s v ov))

(def observe (o i s v ov)
  (if (acons o)
      (map (fn (o2) (o2 i s v ov)) o)
    o (o i s v ov)))

;;; --- constructor sugar --------------------

(mac imd (name (type) . inits)
  `(inst ',type 'name ',name
     ,@(mappend (fn ((s v)) `(',s ,v)) (pair inits))))

(def c-in (v)
  (inst 'cell 'cache v))

(mac c? (rule . observers)
  `(inst 'cell
     'rule ,rule
     'observers (list ,@observers)))

;;; --- example --------------------------------

(defmd (furnace (model) fur-)
  on temp (fuel 0)
  ;;; another way to do observers, at the class level
  ;;;  observers `((fuel ,(fn (i s v ov)
  ;;;                       (prt 'md-defined-observer-sees i!name s v ov))))
  )

(defmd (thermostat (model) thermo-)
    preferred actual)

(def test-furnace ()
  (do (cells-reset)
      (prt '----------start-------------------)
    (let (th f) (to-be
                 (list
                  (imd th42 (thermostat) preferred (c-in 70) actual 70)
                  (imd f-1 (furnace) 
                    fuel 10
                    on (c? [let th (md? 'th42)
                             (< (thermo-actual th)(thermo-preferred th))]
                         ; an instance-level observer
                         (fn (i s v ov)
                           (prt "Sending"(if v 'on 'off) "control sequence to furnace f-1"))))))
      ;;; A global observer of any slot called "on"
      ;;;      (push (fn (i s v ov)
      ;;;              (prt 'on-global-obs-1 i!name s v ov))
      ;;;        obs*!on)
      
      (prt "After awakening the model the furnace is" (if (fur-on f) 'on 'off))
      (set-thermo-preferred th 72) ;; the furnace comes on cuz we want it warmer
      )))

(test-furnace)

;;; Output:
; ----------start-------------------
; Sending off control sequence to furnace f-1
; After awakening the model the furnace is off
; Sending on control sequence to furnace f-1

A Terrible Thing

2008-02-18T00:07:00.004-05:00

some comp.lang.lisp yobbo wrote:
You know, with you, I just don't know how to read your posts (I mean all the stuff about arc). You're like one of those oriental mysteries that this sub-linear mind just cannot fathom.

Google Adsense feels your pain. This is a Lisp blog and I have seen ads for furnace installations and nursing jobs in Hawaii. Well, what do you want from people who use Python?

Lemme help. In your country, do you have the game "in plain view"? The inventor was a genius of psychology, especially perception. Here in Umaguhwallaweeland we like to do it with chocolate candies on Easter aka find-the-candy-day. We take little candies (or for the adults splits of vodka) and scatter them around the house in plain view of anyone standing in any normal location.

The seekers run about frantically freaking out over the fact that they know they are missing chocolates or Stolys just there for the taking while the hiders try not to lose bladder control looking straight at the candies watching the seekers burn more calories than a seagull in a headwind.

Something like this:: Maybe I was too hard on Sohail

That said, your confession indicates you are almost ready to leave the temple. You just need to make the Giant Kenny Leap (GKL). This was when I realized that when I had these weird feelings of dissonance that G*d was speaking to me so I would never be wrong if I just listened, that the dissonance meant something and that for the love of G*d I should not ignore the dissonance.

When writing code, that is, but I recognize the rule in your inner conflict.

Bullseye. Arrow. Just let go.

The GKL moment came working on The Greatest Lisp Application ever now sadly lost to mankind (another story) but during whose creation I realized nothing more sophisticated than that if it feels bad stop it and if it feels good yadda yadda. As you in your temple-leavingness detected this is not and should not be subjected to the impotence of conscious reason, the mistake that doomed AI but I digress.

One of the quotations in my fortune cookie file covers someone who came close but in the end failed on one of my he-cannot-be-serious posts and decided I was serious... when your pattern recognition system recognizes irony, why are you are arguing? Your PRS safely jaywalks you across a dimly lit four-lane highway in a driving rainstorm while it is a crapshoot whether that is deoderant or hairspray your cortex is applying to your underarm.

Just Listen to these vagosities percolating up from neural regions unknown--there is no truer guide. Shakespeare almost had it right:

What a tangled web we weave when first we reason.

And the United Negro College Fund was just a hair outside as well:

The mind is a terrible thing.

Oops, here comes The Odd Couple re-run, gotta go.

Faux dynamic binding for Arc

2008-02-15T03:10:00.005-05:00

The objective in what follows is to simulate one of my favorite features of CL, which is dynamic binding of special variables. If you ever programmed QuickDraw for the Mac and tediously had to code up GetPort, SetPort, and then a SetPort back when finished drawing, just imagine having the ability simply to code:


  (let grafport* (grafPort my-window)
      ...)

...and get the whole save, re-bind, and restore taken care of for you. Does not come up a lot, but when it does this rocks (and Cells uses it for its core behavior of tracking dependencies). You see this sort of thing especially in cases exactly like the drawing GrafPort, something so fundamental that the lowest level functions will need it so it is either a global of some kind or every function in the API needs it as a parameter.

In the case of Cells, without dynamic binding almost every function in the application would need it. Not a pretty thought.

Caveat emptor: The following has been lightly tested with only one variable pseudo-dynamically bound. Bug reports welcome, but if the report is "Listen, dummy, Arc already has that."...well, I'll just be a normallispweeny for a week:


(mac withs* (parms . body)
  (let uparms (map1 [cons (uniq) _] (pair parms))
    `(do ,@(map1 (fn ((save curr val))
                   `(= ,save ,curr ,curr ,val)) uparms)
       (do1
           (do ,@body)
         ,@(map1 (fn ((save curr val))
                   `(= ,curr ,save)) uparms)))))

(= stak* (list 'top))

(def pstack (k)
  (prs k stak*)(prn))

(do
  (pstack 'start)
  (with* (stak* (cons 'me stak*))
      (prt 'with-me stak*)
      (pstack 'confirm-with-me))
  (pstack 'bye-bye))

Output should be:


  start (top)
  with-me (me top)
  confirm-with-me (me top)
  bye-bye (top)

Enjoy. (And with* left as an exercise. :)

Meaningless First Impressions of Arc

2008-02-14T18:41:00.004-05:00

Almost posted on comp.lang.lisp:

> I think porting Cells is a very good test of Arc as a practical
> language (even at this early stage),

Yep, but it occurred to me that Mr. Graham would protest that transliterating a project however interesting is about as far as one can get from exploratory programming and he would be right. Hell, even someone working from a system spec still has to come up with the code.

> I appreciate you jumping in
> as the guinea pig.

I must confess, it is more like guinea frog (inside cll joke). But this dry material probably will not make me the next Imus in the Morning.

> I think it's much more informative than the "I
> wrote a wiki in 10 lines" that we've been seeing. What's your
> impression so far? Is there anything that really stands out, good or
> bad, while you've been working on the Cells port?

With 1% of the polls reporting, I miss my mud (the forty-six page index of Common Lisp the Language). No matter whether I reach for the #3 Torx driver or the articulating socket wrench extender I come up with the same vice-grip. I doubt I will ever get over this.

As for the brevity, that feels more like friction than grease, and I cannot even touch type.

But most of all I feel like I was right: it will be hard to tell until I have done a lot more Arc coding, because the brevity might feel like friction only because I have not gotten used to it. Especially fun is stuff that newly appeared in Arc 1 (click only to download) such as a hashtable lookup normally coded thus:

 (my-table sought-key)

...which can now be coded thus:

 my-table.sought-key

...and when the key is a literal we can rewrite:

  (oed 'peloria) as oed!peloria

This notation is generally applicable where one has a one-parameter function:

  abs.-42 -> 42

So the potential for code obfuscation is terrific. :) To get Cells to work I will have to write readers and writers, so you might see code like:

 (pressure my-boiler)

...which could as well be:

 pressure.my-boiler

...and since the boiler will be a hashtable, one would normally read the hashtable like this:

 my-boiler!pressure

My head is spinning. With Cells that would be a backdoor which would bypass the Cells mechanism entirely a la slot-value in Common Lisp. Bad client! Bad!

If Lisp is a local maximum in language design, all-sexprs-all-the-time might be the knife-edge ridge on which it stands. See we will.

Arc!Cells: Baby Steps

2008-02-14T05:19:00.013-05:00

[This first article on doing Cells (see the "Cells Manifesto" entry in this blog) in Arc serves more to motivate the idea than anything else. In a nutshell, we want to be able to do something like this:


(= a (ci 3))
(= b (ci 4))
(defobserver c ()
    (prs "Hey, C is now" new-value
         "was" old-value))
(= c (c? (sqrt (+ (expt (cval a) 2)
                  (expt (cval b) 2)))))
Hey, C is now 5 was unbound
(cset a 6 b 8)
Hey, C is now 10 was 5

In other words, values are not just functions of values but also active bits of state which can have arbitrary observer logic bound to them.

In a subsequent (next?) blog I plan to easy up on the tutorial angle and just present a decent port of what I called Cells II, which was powerful but had theoretical holes closed in Cells3.

A close version of what follows is in CVS here (as cells-1.arc in case this link does not fly):


   http://common-lisp.net/cgi-bin/viewcvs.cgi/kennysarc2
                /cells-1.arc?rev=1.1&root=cells&view=log

Enjoy.]

First a couple of utilities:


(def prt args
  (apply prs args)
  (prn))

(mac prun (banner . forms)
  `(do (prn ,banner)
       (prn '(do ,@forms))
     (prn)
     ,@forms))

Let's start with the moral equivalent of a C++ member function, or at least I think that's what they call it. Something that looks like a stored data member or slot of an attribute of an object but is in fact implemented as a function of that object.

An example would be having a rectangle object with data members where one could store length and width and then have an area attribute implemented (in C++) as:


   area = this.length * this.width;

Aside from saving a little memory, one gets a guarantee that the area will always be consistent with the length and width, which is not the case if one is writing code that says oh gosh I just changed the length I better go change the area.

As our 'application pushing down on the core' we'll use my favorite, a furnace boiler.


  (= b* (obj outside-temp 72
         on? [< _!outside-temp 50])))

No, the outside temp is not an attribute of a boiler, we're just keeping things in one table as a convenience until we get the ball rolling, later on we'll deal with multiple objects.

That anonymous function above boils down to:


   If the outside temp is less than 50,
     then turn on the boiler,
     otherwise turn it off.

First, let's see if the rule works (not a big accomplishment)


  (prt 'boiler b*!outside-temp (if (b*!on? b*) 'on 'off)))

  -> boiler 72 off

Good, now change temp to 32 and see if the boiler comes on:


  (= b*!outside-temp 32)
  (prt 'boiler b*!outside-temp (if (b*!on? b*) 'on 'off)))

  -> boiler 32 on

Super. Now let's hide the fact that on? is a function
behind a reader function:


  (def on? (i) (i!on? i)))

...and ease inspection:


(def pr-boiler (b)
  (prt 'boiler 'temp b*!outside-temp (if (on? b) 'on 'off)))

Test new slot reader, setting temp high enough this time
so that the boiler should go off:


  (= b*!outside-temp 80)
  (pr-boiler b*))

  -> boiler temp 80 off

But we want more flexibility than having an attribute always defined by a function. Maybe we just want to store nil or t in on? and maintain it as usual, via assignment. Now on? can no longer be assumed to be a function. Fortunately we already have it behind a reader in our burgeoning little OO system, so we just need to enhance that (and get a redefinition warning):


(def on? (i)
  (awhen i!on?
    (if (isa it 'fn)
        (it i)
      it)))

Just test that our new accessor for on?:


  (= b* (obj outside-temp -10
             on? nil)) ;; we'll ignore the outside-temp for this test
  (pr-boiler b*))

  boiler temp -10 off

Now assign t to on?:


  (= b*!on? t) ; We'll hide the assignment implementation later.
  (pr-boiler b*))

  boiler temp -10 on

Good. We will want all our attributes to work this way, so we may as well generalize the on? behavior now:


(def slot-value (i slot-name) ;; i is like self ala Smalltalk
   (awhen i.slot-name
      (if (isa it 'fn)
          (it i)
        it))))

(mac defslot (name)
  `(def ,name (i) (slot-value i ',name)))

(defslot outside-temp)
(defslot on?)
(defslot inside-temp) ;; Let's start elaborating the model

(def pr-boiler (i)
  (prt 'boiler
    'outside-temp (outside-temp i)
    (if (on? i) 'on 'off)
    'inside-temp (inside-temp i)))

And test:


(= b* (obj outside-temp 20
         on? nil
         inside-temp [if (on? _)
                         72
                       _!outside-temp])))

boiler outside-temp 20 off inside-temp 20

Now let's bring back the automatic boiler:


  (= b*!on? [< _!outside-temp 50]))

...and step the outside temperature up from freezing to torrid.


  (loop (= b*!outside-temp 30)
        (< b*!outside-temp 100)
        (= b*!outside-temp (+ b*!outside-temp 10))
     (pr-boiler b*)))

Looks like we need an air conditioner. And let's get more realistic about the model


(= outside* (obj temp 20))
(defslot temp)

(= furnace* (obj on? [< (temp outside*) 50]))
(= ac* (obj on? [> (temp outside*) 75])) ;; air conditioner
(= inside* [if (on? furnace*) 72
             (on? ac*) 68
             (temp outside*)])

(def dumpworld ()
  (prt "outside" (temp outside*))
  (prt "furnace" (if (on? furnace*) 'on 'off))
  (prt "a/c" (if (on? ac*) 'on 'off))
  (prt "inside" (temp inside*)))

Step temperature up from freezing to torrid, but with an air-conditioner:


(loop (= outside*!temp 30)
      (< outside*!temp 100)
      (= outside*!temp (+ outside*!temp 10))
   (prn)
   (dumpworld)))

Nice. We have built a working model that runs by itself given simple declarative rules, meaning we state the rules and an engine sees to it that the model runs. But we have a problem. Let's add a debug option to our slots:


(def slot-value (i slot-name (o debug))
    (awhen i.slot-name
      (if (isa it 'fn)
          (do
              (when debug (prt "Running the rule for slot" slot-name))
              (let result (it i)
                (when debug (prt "...slot" slot-name "is" result))
                result))
        it)))

(mac defslot (name (o debug))
  `(def ,name (i) (slot-value i ',name ,debug)))

(defslot on? t)

Same test tracing the on? slots


(loop (= outside*!temp 30)
      (< outside*!temp 100)
      (= outside*!temp (+ outside*!temp 10))
   (prn)
   (dumpworld)))

Looks OK, but watch what happens even if nothing is going on:


(dumpworld))

Ah, the downside of the functional paradigm: the code runs and runs. For simple functions that is no problem, but if we build an entire application this way things bog down (we learned the usual way).

What we need to cache the result of a calculation and then return the cached result when queried a second time. But then there is a new problem: when do we refresh the cache? Answer: when we have to in order to stay current with the changing world arounds us, more plainly when one of the values used in a calculation changes.

So we need to keep track of who uses whom in their calculations, and when one value changes notify its users that they need to recalculate.

Tomorrow.

Maybe I Was Too Hard On Sohail

2008-02-13T16:56:00.006-05:00

Background: Sohail tried to contradict an article of mine on comp.lang.lisp in which I compared Arc unfavorably to Common Lisp with some code proving CL was almost as good as Arc. Think about that for a second. I ripped him a new one for poor reading comprehension and guessed correctly (he confirmed) that his understanding was blinded by another mistake, misconstruing my impatience with Arc's detractors as meaning somehow a preference of Arc over CL.

Great Lisp-NYC chug-a-lug last night. But... everyone who came up to me to talk about Arc said, "So you like Arc, eh?" Every last one. And most of those who did are smarter than I.

Each time I responded, "No, I never said that. I just rag on people for ragging on Arc without trying it."

It gets worse. Half the people responded they were in fact aware that I had merely ragged on naysayers for their unreasoned opining. And they still thought I liked Arc.

Meanwhile over on Reddit someone wants to know if I am gay because I whined publicly about Turing being persecuted for being gay. Sure and I am also black, Comanche, Jewish, Palestinian, Mexican, an Iraqi civilian, and I died in the plane that hit the South Tower of the WTC.

We all know everyone sees the world differently. What I realized last night is that the differences go beyond the differences in the observers themselves, their different tastes, upbringings, cultures, and DNA. People just plain get things wrong.

We get things wrong all the time. As a matter of course. Our world views form a scatter plot around the bullseye of reality.

Not because of lack of intellect or through lack of attention, though those explain the worst misses. Just because the brain registers new information using a sloppy indexing scheme that settles for bad matches because they are the closest matches and because those matches are affected by what we had for lunch.

We need a new song. Maybe "I say potato, you say Hunh? More day-glo?".

One of my favorite lines to deliver upon completing a task: "That's close enough for government work." I think the brain's ethic is "That's close enough for this bozo carrying me around."

Super. Billions of us walking around trying to deal with each other while each of us is working from more or less faulty copies of reality. Explains a lot.

I realize now I never bought having conflict and injustice explained as the natural consequence of different tastes and life experiences. Nope. We are all just more or less wrong about more or less everything, each of us in different ways, and so no, Rodney, we can never just get along. We will always be in conflict because we will never agree because we are all carrying around different broken realities and broken realities will clash more violently than realities varying merely by taste.

Why do I find this so comforting?

A Tall Ship and a Star to Steer Her By

2008-02-11T06:48:00.001-05:00

A few things collided yesterday. Background: I slammed Scheme and academia for the star by which they chose to steer Scheme's design, namely purity and a small specification. Yep, you heard me right. A design goal was a small spec, nothing to do with making programming easier. From the FAQ:

Advocates of Scheme often find it amusing that the Scheme standard is shorter than the index to CLtL2.

Amuse this. Anyway, after that I got yelled at for being mean to Scheme and academia. Hey, I thought that was what blogs were for.

Now the other thing that happened. Paul Graham released a new Lisp called Arc for which his guiding star is the brevity of the *application code* written in Arc. (He also likes a language to be small but I cannot be mean to him because he once was nice enough to give me a tip on how my RoboCup team could play better defense.)

Mr. Graham further mentioned that Arc development includes a Web app library because he wants there to be a real-world application "pushing down" on the core language design. This reminds me of my cherished Cells library whose twelve-year evolution has always been pushed by application development mostly because I am just a simple application programmer not smart enough to come up with fancy theories in the abstract so I just use things till they break and then fix them just well enough to get them working again. But I digress.

The idea of having an application shape Arc went down well -- no one whined about it and on Usenet that is tantamount to a standing ovation -- but Mr. Graham's choice of brevity as a pole star in navigating his Arc led to a raucous of blogwar and I still had that defender of academia challenging me with "Hey, McCarthy was an academic". Things got ugly, the lights went out, I took a chair over the head, and here is what we (I) decided.

First principles are Good Things. In Lisp, every form returns a value. That first principle works great. This despite Tilton's Law of Programming:

All X all the time is the root of all evil.

Examples: Prolog where it is all logic and unification and Smalltalk and Java where it is all objects. So how did "every form returns a value" get past Tilton's Law? Beats me, must be a Deep Truth somewhere about the power of the functional paradigm which is not all that hard to understand if one has worked on even a small amount of stateful code. At any rate, we will not reject out of hand the idea of one idea shaping a language, but we do see that we gots to pick the right idea.

Next question: are we OK with shorter being better, all other things being equal? Yes, that is correct. Note that reductio adsurbum counterexamples such as APL and K and Perl do not work because in those cases the shorterness takes us out of the space of all other things being equal. Changing Lisp's lambda to Arc's fn makes it shorter without making it obscurer, but changing lambda to a hieroglyphic would.

Next issue: the shorter specs of Scheme and Arc. Would you walk up to a working programmer and start jumping up and down about this amazing new language guess what it's spec fits on one page!!! The working programmer would worry about you. Yeah it makes the compiler easier to write but do we working programmers tell you compiler writers our problems?

Small specs. Pfft. Ever walk into an auto garage and see one of those huge red chests filled with drawer after drawer of tools? Those chests are the seven years of med school for a mechanic. We civilians might think it an overwhelming selection but the mechanic knows it by heart, even those tools used like every two months.

Mr. Graham talked at one point about a language being small enough to keep in one's head. I think the mistake there is the assessment of how big that small can be. People using something a lot (and I hope we are talking about designing a language for someone using it every day) can keep quite a bit in their heads. Meanwhile, hey, these are computers, a good IDE lets us zero in on documentation with a minimum of keystrokes to get the details of a keyword we forget, including zero keystrokes cuz as I finish typing the name of a function my IDE status bar shows the expected parameter list.

On the flip side of big, a scanty starting tool chest means everyone either reinvents the missing bits or grabs one or another incompatible library off the shelf which is why Scheme programmers cannot share their work. Some of them point to specific implementations such as DrScheme and say Hey, it has everything! in which case what was the point of the small standard? You now have a big language unlike anyone else's. Juusst peachy.

All that has been achieved with a small starting standard is making my code incompatible with someone else using some other big Scheme implementation. Lisp went through that without trying to because it too started small, insanely small, but then they cured the natural outcome (a fragmented community) when that threatened its DARPA contracts by creating Common Lisp the standard and everyone saluted and conformed. They still lost the DARPA contracts but those will be back eventually because the standard now has the Common Lisp community all pulling in the same direction: I can use Other People's Code and they can use mine.

The subtext here is that CL may be a big language but I use all of it (except series, but give me time, it has only been thirteen years). One of the classic auto mechanic sayings is that any given task is easy when you have the right tool, and all of us shade tree mechanics forced into retirement by ever more complex automobiles remember how great were those moments when someone handed us the right tool and we were able to put down the Vice-Grips.

Brevity of application code? Great! Brevity of spec, not so much. Scheme needs to close up shop because if they fix what is wrong with it (hygienic macros, NIL not being false, a tiny standard) they end up with CL.

Arc is differentiated by its use of syntax to yield shorter applications, so it can stay, but it does need to add some drawers to the tool chest. And academia needs to start writing applications at the same time they are having bright ideas so they stop designing cool things that happen to be useless for programming. More on constraints later.

What the Hell is a Fortune Cookie File Anyway?

2008-02-11T03:31:00.005-05:00

Just to get strangers up to speed quickly on your correspondent, here is what other people say I said, according to their sigs:

%
I think it would be harder for a good programmer to change editors than to change languages.
%
Parentheses? What parentheses? I haven't noticed any parentheses
since my first month of Lisp programming. I like to ask people who
complain about parentheses in Lisp if they are bothered by all the
spaces between words in a newspaper...
%
Lisp gives you a kazillion ways to solve a problem.
(1- kazillion) of them are wrong.
%
My dream of a Cells consulting business is fading as user after user
turns out to be hindered not at all by my deliberate withholding
of documentation.
%
Lisp nearing the age of fifty is the most modern language out there. GC, dynamic, reflective, the best OO model extant including GFs, procedural macros, and the only thing old-fashioned about it is that it is compiled and fast.
%
Eight years to do TeX? How smart can [Knuth] be? He should have used Lisp.
%
[This one apparently stopped the quoter from trying Lisp. The quoter suspected irony but finally decided I was serious.]
Yeah, I'm a gifted guru. Since you called me that, I guess I'll talk to you a little bit.
%
There are no average Lisp programmers. We are the Priesthood. Offerings of incense or cash will do.
%
I suppose if what you said had any merit it would occasion hostility.
%
Yeah, what the hell can anyone accomplish with beauty and power? (in response to a linked blog entry saying Lisp was beautiful and powerful but unuseable)
%
No, you are an asshole doing his best to despoil c.l.l, and I am an
asshole doing his best to hose out your crap.
%
Liars need good memories, trolls need NG readers with bad ones.
%
We all need early warning systems for "they might have heard this a million times before". I once destroyed any nano-chance I had with a stunning, taller woman by asking her how tall she was.
"Six foot two," she replied. "How short are you?"
%
Programmers who lock onto a design decision and cling to it in the face of contradictory new information -- well, that's almost everyone in my experience, so I better not say what I think of them or people will start saying bad things about me on c.l.l.
%
This reminds me of the NYC cabby who accepted a fare to Chicago. When they got there and could not find the friend who was supposed to pay the fare the cabby just laughed and said he should have known.
%
> Actually, I believe that Aikido, Jazz and Lisp are different appearances of the same thing.
Yes, the Tao. /Everything/ is a different appearance of the tao.
---
"Ken, I went to the library and read up on Buddhism, and believe me, you are no Buddhist."
-- Kenny's Mom
%
That absolutely terrifies the herd-following, lockstep-marching, mainstream-saluting cowards who obediently dash out to scoop up books on The Latest Thing. They learn and use atrocities like Java, C++, XML, and even Python for the security it gives them and then sit there slaving away miserably, tediously, joylessly paying off mortgages and supporting ungrateful teenagers who despise them, only to look out the double-sealed thermo-pane windows of their central-heated, sound-proofed, dead-bolted, suffocating little nests into the howling gale thinking "what do they know that I do not know?" when they see us under a lean-to hunched over our laptops to shield them from the rain laughing our asses off as we write great code between bong hits.... what was the question?
%
Shut up! (That last phrase has four or more syllables if pronounced as intended.)
%
Nonsense. You'll be using it for the GUI, not protein-folding.
(in response to a comment that LTk was slow)
%
Continuations certainly are clever, but if we learned anything from the rejection of the cover art for "Smell the Glove", it is that "there is a fine line between stupid... and clever".
%
Ah, there's no place like academia for dispassionate, intellectually honest discussion of new ideas on their merits. Thank god for tenure giving your bold antagonist the protection they needed to shout down your iconoclastic..... hang on...
%
Whoever objected must be in my killfile, ...
%
From memory (but I think I have it right): "But Jesus said, Suffer captured variables, and forbid them not to come unto thine macro bodies, for of such are DSLs made." Can I get an Amen?
%
Awareness of defect is the first step to recovery.
%
You made a bad analogy (there are no good ones, but you found a new low) ...
%
Yes, it is true that Kent Pitman was raised by a closet full of Lisp Machines, but the exception only proves the rule. (in a postscript after positing that computer languages are not learned in infancy)
%
I suggest you try bartender's school to support yourself, start programming for fun again.
(responding to a comment that 98% of anything to do
with computers was not interesting code)
%
I could add four lanes to my carpal tunnel and still not write all the code I am dying to write.
%
Neutrality? I want to bury other languages, not have a gateway to them.
%
Ken: "Cute puppy. Did you get it for companionship or to pick up chicks?"
Simon: "Hunh? My puppy /always/ gives me companionship."
(on how he was understood by a native English speaker)
%
> Yeah. In case you guys didn't quite get it, this particular site
> targets (among other things) the Turkish lisp effort. Yours truly and
> others have also been accused of being destructive to Turkish youth
They want you to copy them first?
%
You can lead a horse away from water, but if it wants to drink you
better have a big frickin stun gun. No, I have no idea what I just said.
%
PWUIAHAHAHAHA. This is Lisp. We never use anyone else's code. Hell, most of us won't even use Lisp, insist on creating our own with C.
%
OK, who is going to head over to c.l.python and suggest parentheses instead of whitespace? It could be a pretty funny post. "Hey, guys! I was looking at Lisp, had this great idea...keep an open mind!"
%
Educational software itself is now an acknowledged joke precisely because of people who try to foist on others "Oh, look, we used a computer, it must be in-frickin-credible".
%
Horse. Cart. Please note order.
%
Start this thread again. Change the subject to "Lisp Sucks! Molasses is faster!!" and post the code and some timings and the whole community will be working for you for free.
%
We live here. Every week we see noobs stumble in the door, trip over the upturned rug, bang their heads on the too-low hanging lamp, and then ease into Lisp pretty easily after being pointed to Lisp-aware editors, Slime, PCL, etc etc directed by living Lisp Gods, dynamic and interactive, not some two-dimensional billboard of an FAQ that just raises more questions.
%
Are you looking for that "friends" stuff from C++ (if I have that right, unless it was Java ). This is Lisp, we just shoot from the hip.
%
You people do not listen to me on AllegroCL, you do not listen to me on Cells, why am I not surprised you do not listen to me on the tao?
%
Two prehensile toes up!
-- reporting on behalf of his development team their review of Peter Seibel's "Practical Common Lisp"
%
Pearls. Swine. Please note order.
%
> I think it's a fairly large assumption that this problem is soluble...
If not work on it should be suspended.
%
This will be a commercial "Hello World" app that displays "Hello", asks you to login or buy, then displays "world". More soon.
%
> Callbacks are a form of continuation.
Yes, in the same sense that a shoe is a form of aircraft carrier.
%
Lisp is no longer the crazy aunt in the attic, she is now out in the front parlor where her admirers come to pay homage and learn at her feet.
%
> fred gilham wrote:
>> kenny tilton wrote:
>>> fred gilham wrote:
>>> Become a celebrity like Kenny Tilton.
>> I am a simple Lisp programmer.
>> --
>> "I am a simple Buddhist monk. "
>> -- Tenzin Gyatso, the Fourteenth Dalai Lama
> On top of everything else, humble too! :-)
%

My Road to Lisp

2008-02-11T03:18:00.001-05:00

I, Kenny Tilton, do solemnly offer these my responses to The Road to Lisp Survey:

When did you first try Lisp seriously?

1995. My initial interest in programming when I went shopping for my first computer in 1977 was artificial intelligence so of course I heard about Lisp,and when I ran across a Lisp 1.5 Manual in some geek shop I snagged it. But I could not make any sense out of it.

Fast forward five years and now I am a VAX/VMS consultant doing business apps in tall buildings, and while reading some geek rag I am stunned to see Lisp listed as a 4GL based on the 4GL definition of being an order of magnitude more productive than a 3GL. This was strange because 4GLs generally created the illusion of greater productivity with screen builders and a super high level hence inflexible language with database stuff done for you and I knew Lisp was not like that. I filed this datapoint away and went on with my life.

Five years later I decide to write a program which could check intermediate steps of arbitrary first-year high school algebra problems as part of a math tutorial. I see Microsoft (!) offers a Logo, and I remember the 4GL datapoint and decide what the hell let's learn a new language at the same time.

I loved it and went amazingly far with Logo on the tutorial before realizing it would have to be ported to C (a most unpleasant exercise) and it was another ten years before I found my way out of the desert and back to Lisp.

Which Lisp did you try?

Common Lisp. Digitool's MCL, to be specific.

What led you to try Lisp?

I was looking for A Better Way than C for version two of my algebra tutorial. I knew I wanted OO because I could see in my C code that I was using structs in an OO kinda way. I also knew I wanted a dynamic, interactive development environment and it had to run on the Mac where I made most of my sales.

The first thing I tried was Stoney Ballard's Component Workshop, a brave attempt at a dynamic C++ which never even got to "Hello world" before Stoney gave up. Then I got Symantec C++ (I had used it when it was Michael Kahl's Think C) and freaked out when I saw incremental linking was gone and every edit-run cycle now included a twenty second link from scratch. And I was just appalled by C++ the language.

Next up was SmalltalkAgents from Quasar Knowledge Systems. QKS kept saying how great they were and how great they were going to be. Perhaps, but not by crashing my Mac every five minutes.

I then announced on Compuserve's MACDEV forum that I was giving up and was going to use C++,when a kind soul from Cambridge, Mass suggested I look at MCL. He said it was compiled, fast, mature, object-oriented and a few other things.

Well the amazing thing is that I had been skipping MCL ads in the Apple Developer association catalog for months because of my earlier experience with Logo. I had no idea of the progress in Lisp compilers.

I also had come incredibly close when I got the pre-alpha Dylan CD from Apple Computer and played with that. Dylan was being developed for Apple by Digitool and it ran atop (wait for it) Macintosh Common Lisp! Who knew?

But having read the message from Cambridge I promptly ordered a copy of MCL. It came a few days later and just while assembling the manual pages into a binder I concluded I had found The One. I gave the install CD to my two developers, told them to dump QKS, then ordered two more copies to stay honest.

The next few weeks involved everyone taking turns running to each others desk to show them neat new features we had just discovered and I have never looked back.

If you were trying Lisp out of unhappiness with another language, what was it and what did you not like about it, or what about Lisp were you hoping to find different?

I wanted a dynamic development modality, no linker especially. And an end to pointers and dereferencing and manual memory management. ie, I was looking for a better way than C and when I looked at C++ I did not find it.

How far have you gotten in your study of Lisp?

Well, it's been thirteen years, so as an application developer I am fluent.

What do you think of Lisp so far?

Perfect. Whatever it does not do, macros let me coax from CL. Being able to restart from a backtrace after hours of debugging or while trying twenty things a minute apart to get something working is an enormous win. CLOS is unspeakably powerful. A simple thing like special variables is magnificent when you really need them. Of course garbage collection is a stupendous win. The speed is crucial. Multiple inheritance and multi-methods are vital.

One thing a lot of newbies experience as did I is that when I go to find out how some function works it always turns out to work just the way I happen at that time to expect or want or need it to work. Clearly Common Lisp is the product of a long history of refinement by good engineers with a strong ethic of doing things well.

And if you have to know, Lawdy, I never want to edit without parentheses again. When reworking code the code tree is perfect for moving chunks of semantics around. And the autoindentation is a significant time saver. I refactor a lot so a lot of time in other languages goes into tediously realigning code.

The Cells Manifesto

2008-02-11T02:02:00.002-05:00

[We may as well get this out of the way, an explanation of my pet project Cells, the kind of thing I'll be sitting in a rocking chair twenty years from now mumbling how the CIA stole it from me but they don't have the latest version I have that in a floppy disk here in my shirt pocket.]

In the text that follows, [xxx] signifies a footnote named "xxx" and
listed alphabetically at the end.

Summary
Cells is a mature, stable extension to CLOS[impl] allowing one to create classes whose instances can have slot values determined by instance-specific formulas. It is useful for any application involving an interesting amount of long-lived state and a stream of unpredictable inputs. Two examples are any GUI-based application and a RoboCup client (trying to simulate a soccer player given a stream of sensory data fed it by the game server).

Example
For example, in a text editor application we might have (condensed, and with the symbol ".w." macroexpanding to code that returns the relevant window)):


 (make-instance 'menu-item
:label "Cut"
:enabled (c? (bwhen (f (focus .w.))
                       (and (typep f 'text-widget)
     (selection-range f)))))

Translated, the enabled state of the Cut menu item follows whether or not the user is focused on a text-edit widget and whether they have in fact selected a range of text, the trick being that nil is treated as false by (decent) Lisp dialects.

Meanwhile, the selection-range rule might be:


(let (start)
 (c? (if (mouse-down? .w.)
         (bwhen (c (mouse-pos-to-char self (mouse-pos .w.)))
           (if start
               (list start c)
             (setf start c)))
       (setf start nil))))

Now the only imperative code needed is some glue reading the OS event loop onverting raw mouse down and mouse move events into window attributes such as mouse-down? and mouse-pos. The desired functionality is achieved by declarative rules which (like selection-range above) are entirely responsible for deciding the selection range.

A final trick comes from slot observers. Suppose we are thinly wrapping a C GUI and need to do something in the C library to actually make menu items available or not. It might look something like this:


(defobserver enabled ((self menu-item) new-value old-value old-value-bound?)
    (menu-item-set (c-ptr self) (if new-value 1 0)))

ie, Some model attributes must be propagated outside the model as they change, and observers are callbacks we can provide to handle change.

Motivation
As a child I watched my father toil at home for hours over paper spreadsheets with pencil and slide rule. After he changed one value, he had to propagate that change to other cells by first remembering which other ones included the changed cell in their computation. Then he had to do the calculations for those, erase, enter...and then repeat that process to propagate those changes in a cascade across the paper.

VisiCalc let my father take the formula he had in mind and put it into (declare it to) the electronic spreadsheet. Then VisiCalc could do the tedious work: recalculating, knowing what to recalculate, and knowing in what order to recalculate.

Cells do for programmers what electronic spreadsheets did for my father. Without Cells, CLOS slots are like cells of a paper spreadsheet. A single key-down event can cause a cascade of change throughout an application. The programmer has to arrange for it all to happen, all in the right order: delete any selected text, insert the new character, re-wrap the text, update the undo mechanism, revisit the menu statuses ("Cut" is no longer enabled), update the scroll bars, possibly scroll the window, flag the file as unsaved...

Here is a real-world case study:

"The last company I worked with made a product that was a control unit for some mechanical devices, presenting both sensor readings coming in from those devices and an interface to program the devices. Consider it like a very sophisticated microwave oven, perhaps with a temperature probe.

"The UI code was a frighteningly complex rat's nest. Input data arriving from the sensors changed certain state values, which caused the display to update, but the system state also changed, and rules had to be evaluated, the outcome of which might be tuning to the running job or warning messages presented to the user, and in the meantime the user may be adjusting the running job. I'm sure there are even more interactions I'm leaving out.

"There was no "large idea" in this code to organize these dependencies or orchestrate the data flow. The individual facilities were well-formed enough: "message" input and output, GUI widgets and forms, real-world entities modeled as entities in the code. However, the connections between these things were ad-hoc and not formalized. Every change to the system would provoke defects, and the failure usually involved not propagating some event, propagating it at the wrong time, or propagating it to the wrong recipients."
--- Steven Harris, on comp.lang.lisp

What Mr. Harris describes is what Fred Brooks [bullet] said was an essential property of software development, meaning by essential that there was no way around it, and thus his prediction that a software silver bullet was in principle impossible.

Which brings us to Cells. See also [axiom] Phillip Eby's developing axiomatic definition he is developing in support of Ryan Forseth's SoC project.

DEFMODEL and Slot types
Classes, some of whose slots may be mediated by Cells, are defined by DEFMODEL, which is exactly like DEFCLASS but adds support for two slot definition options, :cell and :unchanged-if. Classes defined by DEFMODEL can inherit from normal CLOS classes.

New slot definition options
:cell {nil | t | :ephemeral}

:cell is optional. The default is ":cell t", meaning the Cells engine will manage the slot to give it the spreadsheet-like characteristics. Specifying NIL signifies that this slot is entirely outside any handling by the Cells engine; it is just a plain CLOS slot.

This next bit will not make sense until we have explained propagation of state change, but specifying :ephemeral causes the Cells engine to reset the apparent slot value to NIL immediately and only after fully propagating any value assumed by the slot, either by assignment to an input Cell (the vastly more common case) or by a rule calculation.

Ephemeral cells are necessary to correctly model events in the otherwise steady-state spreadsheet paradigm.

:unchanged-if

Specifying :unchanged-if is optional. [Come to think of it, it should be an error to specify both :cell nil and :unchanged-if.] If specified, the named function is a predicate of two arguments, the new and old value in that order. The predicate determines if a subsequent slot value (either computed or assigned to an input) is unchanged in the sense that no propagation is necessary, either to dependent ruled cells or (getting ahead of ourselves again) "on change" observers.
The default unchanged test is EQL.

Cell types
The Cells library allows the programmer to specify at make-instance time that a Cell slot of an instance be mediated for the life of that instance by one of:

-- a so-called "input" Cell;
-- a "ruled" Cell; or
-- no Cell at all.

Note that different instances of the same class may do different things Cells-wise with the same slot. One label widget may have a fixed width of 42 and text "Hi, Mom!", where another might have an input Cell mediating the text (so edit logic can assign new values as the user types) and a rule mediating the width so the widget can have a minimum width of 42(so it does not disappear altogether) yet grow based on text length and relevant font metrics to always leave room for one more character (if the GUI design calls for that).

To summarize, the class specification supplied with DEFMODEL specifies whether a slot can /ever/ be managed by the Cells engine. For those that can, at and only at instance initialization time different instances can have different Cell types and rules specified to mediate the same slot.

Input Cells
A slot mediated by an input Cell may be assigned new values at runtime. These are how Cell-based models get data from the world outside the model -- it cannot be rules all the way down. Typically, these input assignements are made by code polling OS events via some GetNextEvent API call, or by callbacks registered with an event system such as win32 WindowProc functions. Other code may poll sockets or serial inputs from an external device.

Ruled Cells
Ruled Cells come with an instance-specific rule in the form of an anonymous function of two variables, the instance owning the slot and the prior value (if any) computed by the rule. These rules consist of arbitrarily complex Common Lisp code, and are invoked immediately after instance initialization (but see the next bit on lazy cells).

When a rule runs, any dynamic read (either expressly in the rule source or during the execution of some function invoked by the rule) of a slot of any instance mediated by a Cell of any type establishes a runtime dependency of the ruled cell on the slot of the instance that was read. Note then that thanks to code branching, dependencies can vary after every rule invocation.

Lazy Ruled Cells
Laziness is cell-specific, applies only to ruled cells, and comes in four varieties:

:once-asked -- this will get evaluated and "observed" on initialization, but then not get reevaluated immediately if dependencies change, rather only when read by application code.

:until-asked -- this does not get evaluated/observed until read by application code, but then it becomes un-lazy, eagerly reevaluated as soon as any dependency changes (not waiting until asked).

:always -- not evaluated/observed until read, and not reevaluated until read after a dependency changes.

Dataflow
When application code assigns a new value to an input Cell (a quick way of saying an instance slot mediated by an input Cell) -- typically by code polling OS events or a socket or an input device -- a cascade of recalculation ensues to bring direct and indirect ruled dependents current with the new value assigned to the input Cell.

No Cell at All
Because of all that, it is an error to assign a new value to a slot of an instance not mediated by any Cell. The Cells engine can do a handy optimization by treating such slots as constants and not creating dependencies when ruled Cells read these. But then we cannot let these Cells vary and still guarantee data integrity, because we no longer know who else to update in light of such variation. The optimization, by the way, extends to eliminating ruled Cells which, after any computation, end up not depending on any other cell.

Again, note that this is different from specifying ":cell nil" for some slot. Here, the Cells engine has been told to manage some slot, but for some instance the slot has been authored to bear some value for the lifetime of that instance.

Observers
To allow the emergent animated data model to operate usefully on the world outside the model--if only to update the screen--programmers may specify so-called observer callbacks dispatched according to: slot name, instance, new value, old value, and whether the old value actually existed (false only on the first go). Observers are inherited according to the rules of CLOS class inheritance. If multiple primary observer methods apply because of inheritance, they all get run, most specific last.

ie, observers are a GF with PROGN method combination.

Observers get called in two circumstances: as part of Model object initialization, in a processing step just after CLOS instance initialization, and when a slot changes value. Any observer of a Cell slot is guaranteed to be called at least once during intialization even if a cell slot is bound to a constant or if it is an input or ruled Cell that never changes value.

It is legal for observer code to assign to input Cells, but (a) special syntax is required to defer execution until the observed state change has fully propagated; and (b) doing so compromises the declarative quality of an application -- one can no longer look to one rule to see how a slot (in this case the input slot being assigned by the observer) gets its value. A reasonable usage might be one with a cycle, where changing slot A requires a change to slot B, and changing slot B requires a change to slot A, such as the scroll thumb position and the amount a document has been scrolled.

Finally, to make it possible for such a declarative model to talk intelligibly to imperative systems such as Tcl/Tk which sometimes requires a precise sequence of commands for something to work at all, a mechanism exists by which client code can (a) queue tasks for execution after a data change has fully propagated and (b) process those tasks with a client-supplied handler. Tasks are queued with arbitrary keying data which can be used by the handler to sort or compress the queued tasks.

Data Integrity
When application code assigns to some input cell X, the Cells engine guarantees:

- recomputation exactly once of all and only state affected by the change to X, directly or indirectly through some intermediate datapoint. note that if A depends on B, and B depends on X, when B gets recalculated it may come up with the same value as before. In this case A is not considered to have been affected by the change to X and will not be recomputed.

- recomputations, when they read other datapoints, must see only values current with the new value of X. Example: if A depends on B and X, and B depends on X, when X changes and A reads B and X to compute a new value, B must return a value recomputed from the new value of X.

- similarly, client observer callbacks must see only values current with the new value of X; and

- a corollary: should a client observer SETF a datapoint Y, all the above must
happen with values current with not just X, but also with the value of Y /prior/
to the change to Y.

- Deferred "client" code must see only values current with X and not any values current with some subsequent change to Y queued by an observer

Benefits
Program state guaranteed to be self-consistent, without programmer effort. Dependencies are identified
by the engine, and change propagation happens automatically.

Greater object re-use. Slots of instances can be authored with rules, not just literal values. In a sense, we get greater reuse by allowing instances to override slot derivations instance by instance. But not slot expressions, which are still class-oriented. By this I mean the observers expressing changes in value are dispatched by the class of the instance and so are not instance-specific. (Such a thing has been suggested, however.) Another strong bit of class-orientation comes from the fact that code reading slot X of some instance Y obviously does so without knowing how the returned value was derived. It knows only that the slot is named X, and will do things with that value assuming only that it has the X attribute of the instance Y. So again: the derivation of a slot value is potentially instance-oriented under Cells, but its expression or manifestation is still class-oriented.

Natural decomposition of overall application complexity into so many simple rules and slot observers. Let's return for a moment to VisiCalc and its descendants. In even the most complex financial spreadsheet model, no one cell rule accesses more than a relatively few other spreadsheet cells (counting a row or column range as one reference). Yet the complex model emerges. All the work of tracking dependencies is handled by the spreadsheet software, which requires no special declaration by the modeller. They simply write the Cell rule. In writing the rule, they are concerned only with the derivation of one datapoint from a population of other datapoints. No effort goes into arranging for the rule to get run at the right time, and certainly no energy is spent worrying about what other cells might be using the authored cell. That cell has certain semantics -- "account balance", perhaps -- and the modeller need only worry about writing a correct, static computation of those semantics.

Same with Cells. :) The only difference is that VisiCalc has one "observer" requirement for all cells: update the screen. In Cells applications, a significant amount of application functionality -- indeed, all its outputs -- end up in cell observers. But as discussed above, this additional burden falls only on the class designer when they decide to add a slot to a class. As instances are created and different rules specified for different slots to achieve custom behavior, the effort is the same as for the VisiCalc user.

Model Building
Everything above could describe one instance of one class defined by DEFMODEL. A real application has multiple instances of multiple classes. So...

-- cells can depend on other cells from any other instance. Since a rule gets passed only "self", Cell users need something like the Family class included with the Cells package effectively to turn a collection of instances into a network searchable by name or type.

-- The overall model population must be maintainable by Cell slots such as the "kids" slot of the Family class. The burden here is on the Cells engine to allow one cell of one child to ask for the value of a cell of another child and vice versa (with different Cells), when both children are the product of the same rule, or different rules when "cousins" are exchanging information. So we must gracefully traverse the parent/kids tree dispatching kids rules just in time to produce the other instance sought.

-- kid-slotting: used almost exclusively so far for orderly GUI layout, a parent must be able to specify rules for specific slots of kids. Example: a "stack" class wants to provide rules for child geometry specifying left, right, or centered alignment and vertical stacking (with optional spacing) one below the other. The idea is that we want to author classes of what might be GUI subcomponents without worrying about how they will be arranged in some container.

-- finalization: when an instance appears in the "old kids" but not in the "new kids", a Cells engine may need to arrange for all Cells to "unsubscribe" from their dependents. Cells takes care of that if one calls "not-to-be" on an instance.

Suggested Applications
Any application that must maintain an interesting, long-lived data model incorporating a stream of unpredictable data. Two examples: any GUI application and a RoboCup soccer client.

An application needing to shadow data between two systems. Examples: a Lisp GUI imlemented by thinly wrapping a C GUI library, where Lisp-land activity must be propagated to the C GUI, and C GUI events must propagate to Lisp-land. See the Cells-Gtk or Celtk projects. Also, a persistent CLOS implementation that must echo CLOS instance data into, say, SQL tables.

Prior Art (in increasing order of priorness (age))
Functional reactive programming:
This looks to be the most active, current, and vibrant subset of folks working on this sort of stuff.
Links:
FlapJax (FRP-powered web apps) http://www.flapjax-lang.org/
http://lambda-the-ultimate.org/node/1771
http://www.haskell.org/frp/
FrTime (scheme FRP implementation, no great links) http://pre.plt-scheme.org/plt/collects/frtime/doc.txt

Adobe Adam, originally developed only to manage complex GUIs. [Adam]

COSI, a class-based Cells-alike used at STSCI in software used to
schedule Hubble telescope viewing time. [COSI]

Garnet's KR: http://www.cs.cmu.edu/~garnet/
Also written in Lisp. Cells looks much like KR, though Cells was
developed in ignorance of KR (or any other prior art). KR has
an astonishing number of backdoors to its constraint
engine, none of which have turned out to be necessary for Cells.

The entire constraint programming field, beginning I guess with Guy Steele's
PhD Thesis in which he develops a constraint programming language or two:
http://portal.acm.org/citation.cfm?id=889490&dl=ACM&coll=ACM
http://www.cs.utk.edu/~bvz/quickplan.html

Sutherland, I. Sketchpad: A Man Machine Graphical Communication System. PhD thesis, MIT, 1963.
Steele himself cites Sketchpad as inexplicably unappreciated prior
art to his Constraints system:

See also:
The spreadsheet paradigm: http://www.cs.utk.edu/~bvz/active-value-spreadsheet.html
The dataflow paradigm: http://en.wikipedia.org/wiki/Dataflow
Frame-based programming
Definitive-programming

Commentary
----------
-- Jack Unrue, comp.lang.lisp
"Cells provides the plumbing for data dependency management which every
non-trivial program must have; a developer using Cells can focus on
computing program state and reacting to state changes, leaving Cells to worry about
how that state is propagated. Cells does this by enabling a declarative
mechanism built via an extension to CLOS, and hence achieves its goal in a way
that meshes well with with typical Common Lisp programming style."

-- Bill Clementson, http://bc.tech.coop/blog/030911.html
"Kenny Tilton has been talking about his Cells implementation on comp.lang.lisp
for some time but I've only just had a look at it over the past few evenings.
It's actually pretty neat. Kenny describes Cells as, conceptually, analogous to
a spreadsheet cell (e.g. -- something in which you can put a value or a formula
and have it updated automatically based on changes in other "cell" values).
Another way of saying this might be that Cells allows you to define classes
whose slots can be dynamically (and automatically) updated and for which
standard observers can be defined that react to changes in those slots."

-- "What is Cells?", Cells-GTk FAQ, http://common-lisp.net/project/cells-gtk/faq.html#q2
"If you are at all familiar with developing moderately complex software that
is operated through a GUI, then you have probably
learned this lesson: Keeping what is presented through the GUI in-sync with what
the user is allowed to do, and in-sync with the computational state of the
program is often tedious, complicated work. .... Cells-GTK helps
with these tasks by providing an abstraction over the details; each of the tasks
just listed can be controlled by (a) formula that specify the value of
attributes of graphic features in the part-subpart declaration (that declaration
is called 'defpart' in cells-gtk); and, (b) formula that specify the value of CLOS slots."

-- Phillip Eby, PyCells and peak.events,
http://www.eby-sarna.com/pipermail/peak/2006-May/002545.html
"What I discovered is quite cool. The Cells system *automatically
discovers* dynamic dependencies, without having to explicitly specify that
X depends on Y, as long as X and Y are both implemented using cell
objects. The system knows when you are computing a value for X, and
registers the fact that Y was read during this computation, thus allowing
it to automatically invalidate the X calculation if Y changes....
Aside from the automatic dependency detection, the cells system has
another trick that is able to significantly reduce the complexity of
event cascades, similar to what I was trying (but failing) to do using
the "scheduled thread" concept in peak.events.
Specifically, the cells system understands how to make event-based updates
orderly and deterministic, in a way that peak.events cannot. It
effectively divides time into "propagation" and "non-propagation"
states. Instead of simply making callbacks whenever a computed value
changes, the system makes orderly updates by queueing invalidated cells for
updating. Also, if you write code that sets a new value imperatively (as
opposed to it being pulled declaratively), the actual set operation is
deferred until all computed cells are up-to-date with the current state of
the universe."

_____________
Uncommentary

-- Josh Marchan, internal company email:

"some obscure technology that only theoretically benefits us.... I'm not even sure what the advantage here would be."

-- Peter Seibel, comp.lang.lisp:
"I couldn't find anything that explained what [Cells] was and why I should care."

-- Alan Crowe, comp.lang.lisp:
"Further confession: I'm bluffing. I've grasped that Cells is
interesting, but I haven't downloaded it yet, and I haven't
checked out how it works or what /exactly/ it does."

_________
Footnotes

[Adam] "Adam is a modeling engine and declarative language for describing constraints and
relationships on a collection of values, typically the parameters to an
application command. When bound to a human interface (HI) Adam provides
the logic that controls the HI behavior. Adam is similar in concept to a spreadsheet
or a forms manager. Values are set and dependent values are recalculated.
Adam provides facilities to resolve interrelated dependencies and to track
those dependencies, beyond what a spreadsheet provides."
http://opensource.adobe.com/group__asl__overview.html#asl_overview_intro_to_adam_and_eve
________
[bullet] This resolves a problem Fred Brooks identified in 1987: ""The essence of a software
entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms,
and invocations of functions... Software systems have orders-of-magnitude more states than
computers do...a scaling-up of a software entity is not merely a repetition of the same elements
in larger sizes; it is necessarily an increase in the number of different elements. In most cases,
the elements interact with each other in some nonlinear fashion, and the complexity of the whole
increases much more than linearly."
-- http://www.virtualschool.edu/mon/SoftwareEngineering/BrooksNoSilverBullet.html
______
[COSI] "The Constraint Sequencing Infrastructure (COSI) is an extension to
the Common Lisp Object System (*(CLOS)) which supports a constraint
based object-oriented programming model. .....

"A constraint is a specialized method which will be automatically
re-run by the COSI infrastructure whenever any of its input values
change. Input values are any of the object attributes that are
accessed by the constraint, and which are therefore assumed to
alter the processing within the constraint.

"Whenever a state change occurs those constraints which depend upon
that state are added to a propagation queue. When the system is
queried a propagation cycle runs ensuring that the state of the
system is consistent with all constraints prior to returning a value."
-- http://www.cliki.net/ACL2/COSI?source
______
[impl] The Cells library as it stands is all about doing interesting things
with slots of CLOS instances, but Cells is not only about CLOS or even Lisp.
One Cells user is known to have mediated a global variable with a Cell, some work
was done on having slots of DEFSTRUCTs mediated by Cells, and ports to C++, Java, and
Python have been explored.

_______
[axiom] Phillip Eby's axiomatic specification of Cells:

Data Pulse Axioms
=================

Overview: updates must be synchronous (all changed cells are updated at
once), consistent (no cell rule sees out of date values), and minimal (only
necessary rules run).

1. Global Update Counter:
There is a global update counter. (Guarantees that there is a
globally-consistent notion of the "time" at which updates occur.)

2. Per-Cell "As Of" Value:
Every cell has a "current-as-of" update count, that is initialized with
a value that is less than the global update count will ever be.

3. Out-of-dateness:
A cell is out of date if its update count is lower than the update
count of any of the cells it depends on.

4. Out-of-date Before:
When a rule-driven cell's value is queried, its rule is only run if the
cell is out of date; otherwise a cached previous value is
returned. (Guarantees that a rule is not run unless its dependencies have
changed since the last time the rule was run.)

5. Up-to-date After:
Once a cell's rule is run (or its value is changed, if it is an input
cell), its update count must be equal to the global update
count. (Guarantees that a rule cannot run more than once per update.)

6. Inputs Move The System Forward
When an input cell changes, it increments the global update count and
stores the new value in its own update count.

Dependency Discovery Axioms
===========================

Overview: cells automatically notice when other cells depend on them, then
notify them at most once if there is a change.

1. Thread-local "current rule cell":
There is a thread-local variable that always contains the cell whose
rule is currently being evaluated in the corresponding thread. This
variable can be empty (e.g. None).

2. "Currentness" Maintenance:
While a cell rule's is being run, the variable described in #1 must be
set to point to the cell whose rule is being run. When the rule is
finished, the variable must be restored to whatever value it had before the
rule began. (Guarantees that cells will be able to tell who is asking for
their values.)

3. Dependency Creation:
When a cell is read, it adds the "currently-being evaluated" cell as a
listener that it will notify of changes.

4. Dependency Creation Order:
New listeners are added only *after* the cell being read has brought
itself up-to-date, and notified any *previous* listeners of the
change. (Ensures that the listening cell does not receive redundant
notification if the listened-to cell has to be brought up-to-date first.)

5. Dependency Minimalism:
A listener should only be added if it does not already present in the
cell's listener collection. (This isn't strictly mandatory, the system
behavior will be correct but inefficient if this requirement isn't met.)

6. Dependency Removal:
Just before a cell's rule is run, it must cease to be a listener for
any other cells. (Guarantees that a dependency from a previous update
cannot trigger an unnecessary repeated calculation.)

7. Dependency Notification
Whenever a cell's value changes (due to a rule change or input change),
it must notify all of its listeners that it has changed, in such a way that
*none* of the listeners are asked to recalculate their value until *all* of
the listeners have first been notified of the change. (This guarantees
that inconsistent views cannot occur.)

7a. Deferred Recalculation
The recalculation of listeners (not the notification of the listeners'
out-of-dateness) must be deferred if a cell's value is currently being
calculated. As soon as there are no cells being calculated, the deferred
recalculations must occur. (This guarantees that in the absence of
circular dependencies, no cell can ask for a value that's in the process of
being calculated.)

8. One-Time Notification Only
A cell's listeners are removed from its listener collection as soon as
they have been notified. In particular, the cell's collection of listeners
must be cleared *before* *any* of the listeners are asked to recalculate
themselves. (This guarantees that listeners reinstated as a side effect of
recalculation will not get a duplicate notification in the current update,
or miss a notification in a future update.)

9. Conversion to Constant
If a cell's rule is run and no dependencies were created, the cell must
become a "constant" cell, and do no further listener additions or
notification, once any necessary notifications to existing listeners are
completed. (That is, if the rule's run changed the cell's value, it must
notify its existing listeners, but then the listener collection must be
cleared -- *again*, in addition to the clearing described in #8.)

10. No Changes During Notification:
It is an error to change an input cell's value while change
notifications are taking place.

11. Weak Notification
Automatically created inter-cell links must not inhibit garbage
collection of either cell. (Technically optional, but very easy to do.)

All Your Parenthesis Are Belong to Us

2008-02-09T20:45:00.001-05:00

About the name. Just google it. You will find two kinds of mention. In one kind the author has never tried Lisp and is afraid of it and is trying to make Lisp advocates feel bad. In the other kind the author has tried Lisp and says no wonder Lispers are so smug.

OK, but why is it the name of this blog? Think of Queer Nation embracing the epithet. I'll have more to say another time about Turing who invented the computer just to prove a theorem, but long before Stonewall he refused to deny his affection for men at a time in my lifetime when England jailed people for whom they liked and got prosecuted for his courage even though he had single-handedly turned back the Nazis by parachuting onto an Enigma factory to steal one of the machines. I think I have that right. So much for courage -- they made him take hormones that gave him titties to stay out of jail and two years later he killed himself -- but I digress.

smuglispweeny you call me? Lessee...

Smug? "marked by excessive complacency or self-satisfaction". Well, excessive does not work because I am if anything insufficiently complacent and self-satisfied about Lisp but that's a quibble and I hate quibbles so sure, color me smug.

Lisp? Duh.

Weeny? Sinking to the level of schoolyard name calling, are we? Super, that level is as far as I got. But you won't get an insult back, I am just standing here with this smug grin. What gives?

Easy. Lisp is that good. Call me anything you like as long as lisp is in there somewhere. No Lisper has any doubt that Lisp is far and away superior to everything else out there so if you call us names we are just gonna get it printed up on T-shirts. Before you argue read The Road to Lisp to hear what other weenies have to say. Quiz Friday. OK, now you can argue. Or download one free trial Lisp or another and maybe learn something. Or...

If you love the bleeding edge and want to join a brand new Lisp community consisting mostly of brand new Lispers, try Arc. Arc is the brandest newest Lisp and brought to us by Paul Graham, a better advocate for Lisp than I. I would link to Paul's specific page on Lisp if Firefox was worth a damn and I could see the navigation icons. Arc is Scheme with two fixes from Common Lisp -- unhygienic macros and NIL as false -- and small doses of syntax in return for denser code and fewer parentheses (but not so few as to lose the tremendous advantage (yep) they offer when thrashing code).

After Arc makes you fall in love with Lisp, take up Common Lisp. Not Scheme, a failed experiment in minimalism and purity and all sorts of qualities useless to Real Programmers, an experiment brought to you by academia -- nuff said?

Common Lisp resources: Paul Graham also wrote not one but two fine books On Lisp. The good news about On Lisp is that it is a free download, the bad news is that it is pretty advanced, save that till week two. Peter Seibel wrote the newest Lisp tome, Practical Common Lisp, comprehensive yet built around real-world projects hence the title. Available in print from Apress.

Your final great resource is Usenet, comp.lang.lisp to be specific. Read it daily once you have switched to Lisp for a continuing education in Common Lisp.

As for this blog, it will be about Lisp, application programming in general, dispatches from the trenches of my current application in particular, my Cells project (in a domain the grownups call functional reactive programming), RDF (my latest crush), and more Lisp because software matters and the world would be a better place with better software and that means Lisp.

Let us pray.